from IPython.core.display import display, HTML
display(HTML("<style>.container { width:80% !important; }</style>"))

Using fastai V2: data pipelining

Datablock API
check that tfms are applied to input/target
labeling func: lambda o: o

Load the saved merged database, and set the seeds. And doublecheck our data is where we expect.

df = pd.read_pickle(f"data/{COMBINED_DF}.pkl")
np.random.seed(3333)
torch.manual_seed(3333)

<torch._C.Generator at 0x7f2fe82706d0>

image_path = L_ROOT/"data"
#image_path = D_ROOT/DBS['zappos']
batch_size = 64

L([image_path/d for d in df.path])

(#11771) [Path('/home/ergonyc/Projects/Project2.0/snkrfinder/data/Shoes/Sneakers and Athletic Shoes/ALDO/8045616.9.jpg'),Path('/home/ergonyc/Projects/Project2.0/snkrfinder/data/Shoes/Sneakers and Athletic Shoes/ALDO/8045632.9.jpg'),Path('/home/ergonyc/Projects/Project2.0/snkrfinder/data/Shoes/Sneakers and Athletic Shoes/ALDO/8045650.3.jpg'),Path('/home/ergonyc/Projects/Project2.0/snkrfinder/data/Shoes/Sneakers and Athletic Shoes/ALDO/8045650.9.jpg'),Path('/home/ergonyc/Projects/Project2.0/snkrfinder/data/Shoes/Sneakers and Athletic Shoes/ALDO/8093302.3.jpg'),Path('/home/ergonyc/Projects/Project2.0/snkrfinder/data/Shoes/Sneakers and Athletic Shoes/ALDO/8093302.396.jpg'),Path('/home/ergonyc/Projects/Project2.0/snkrfinder/data/Shoes/Sneakers and Athletic Shoes/ALDO/8093302.4854.jpg'),Path('/home/ergonyc/Projects/Project2.0/snkrfinder/data/Shoes/Sneakers and Athletic Shoes/ALDO/8093302.9.jpg'),Path('/home/ergonyc/Projects/Project2.0/snkrfinder/data/Shoes/Sneakers and Athletic Shoes/ALDO/8093326.3.jpg'),Path('/home/ergonyc/Projects/Project2.0/snkrfinder/data/Shoes/Sneakers and Athletic Shoes/ALDO/8093326.9.jpg')...]

df = prep_df_for_datablocks(df)

FastAI "block" for autoencoder

Next we need our own version of ReadTabBatch that will return our inputs

Don't forget to set n_inp=1. Otherwise the default to make the input to 1-len(blocks). Also note that the FeatsResize is used to avoid the random jittering from resize during training. Only the very narrow batch augmentations will be used.

# block.summary(df)

creating the VAE

Variational Auto-Encoder for fastai

I'm going to use a generic convolutional net as the basis of the encoder, and its reverse as the decoder. This is a proof of concept for using the fastai framework, and will experiment with pre-trained resnet and MobileNet_v2 later. I'd like to use the MobileNet_v2 as a direct link ot the "SneakerFinder" tool which motivated this experiment. [see SneakerFinder]

A variational "module" will sit between the encoder and decoder as the "Bottleneck". The Bottleneck will map the resnet features into a latent space (e.g. ~100 dimensions) represented of standard normal variables. The "reparameterization trick" will sample from this space and the "decoder" will generate images.

Finally a simple "decoder" will sample from the variational latents space and be trained to reconstruct the images.

The intention is the latent space can be used to generate novel sneaker images.

AE: deterministic AutoEncoder ( non- variational)

Although we give up the original utility we are going for -- creating new sneakers via the latent space -- having otherwise equivalent non-variational autoencoders for reference will be great. Furthermore, this latent space representation will be amenable to a MMD regularization later on. This will be useful to avoid some of the limitiations of the KLD as a regularizer (overestimation of variance, and some degenerate convergences). Its sort of hack-ey but keeping the tooling equivalent to the betaVAE will ultimatel give some advantages.

It is convenient to avoid the class wrappers to simplify the param splitting for freezing the pre-trained arcitecture.
We could enumerate the class layers and return sequential, but simply making some functions to put the layers togeter is better.

Loss

These inherit from Module. (fastai's wrapper on nn.Module)

Metrics

These are called by the recorder callback and collect quantities to track training. I made a simple meta class so I don't have to repreat the reset and mean value property for all the metrics.

helper for computing the KL Divergence as a function and a `Module`

The AnnealedLoss Callback basically injects a kl_weight parameter into the loss so we can start training without the full KLD regularization for the beta-VAE version.

The fastai Learner class does the training loop. It took me a little digging into the code to figure out how Metrics are called since its not really stated anywhere in the documentation (Note: create PR for fastai for extra documentation on Metrics logic). By default one of the key Callbacks is the Recorder. It prints out the training summary at each epoch (via ProgressCallBack) and collects all the Metrics. Which by default only loss is a train_met and others are valid_met. The Recorder resets (maps reset() to all mets) the metrics before_train and before_valid. The Recorder maps accumulate() to the metrics on after_batch. Finally

AnnealedLossCallback will inject the latent mu and logvar and a kl_weight variable into our loss. The mu and logvar will be used to compute the KLD. The kl_weight is a scheduled weighting for the KLD. You can see the schedule graph of the parameter. At the beginning it will be 0, thus the KLD part of the loss will get ignored. So during 10% of training, we will fit a normal auto-encoder. Then gradually for 30% of trainning, increase kl_weight to 1 and then remain there for the remaining training time so that the auto encoder now becomes full variational. The way this callback is done, the loss will receive this parameter, but not the model.

n_epochs = 10
f_init = combine_scheds([.1, .7, .2], [SchedNo(0,0),SchedCos(0,1), SchedNo(1,1)])
# f = combine_scheds([.8, .2], [SchedCos(0,0), SchedCos(0,.5)])
p = torch.linspace(0.,1.,100)
pp = torch.linspace(0.,1.*n_epochs,100)

plt.plot(pp,[f_init(o) for o in p])

[<matplotlib.lines.Line2D at 0x7f2eecaa6130>]

WARNING: Avoid using early stopping because the AnnealedLossCallback will make the loss grow once the KL divergence weight kicks in.

I want to note something here that was a little confusing to me: params(model) is a builtin fastai PyTorch.core function which returns all of the parameters of the modules. i.e.

def params(m):
    "Return all parameters of `m`"
    return [p for p in m.parameters()]

The toplevel fastai core functions with simple names that almost match class attributes was one of my biggest stumbling blocks in getting acquainted with the fastai v2 API. (The other is the documentation which is autogenerated by the fastdev frameworks from their development noteboooks. More on that struggle and my tips if that is troblesome for you later (here). {% include note.html content='that it is crucial that you don’t freeze the batch norm layers. The bn_splitter collects out all the batchnorm layers. The simple splitting we do only freezes the encoder and leaves the latent layers (i.e. VAE or linear encoding bottlenedck) and the decoder in a parameter group with the batchnorm layers.' %}

parameter `Splitters`

{% include warning.html content='there are two completely different splitters in the FastAI API. This splitter groups the model parameters into groups for freezing and for progressive learning rates. (The other one is splits data into train and validate. I got imminiently confused when I first started with the API by this.' %}

other encoder and decoder types

MobileNet_v2 as the encoder, as a continuation of the original Sneaker Finder

simple bowtie convolutional encoder / decoder (Mimics the GOAT medium blog)

Architecture Hyperparameters:- Latent Size (research default 256, production default 32) - Filter Factor Size (research default 16, production default 32)

Latent Linear Hidden Layer Size (research default 2048, production default 1024)

The encoder architecture is as follows with research defaults from above:

Input 3x128x128 (conv2d block [conv2d, batchnorm2d, relu])

16x64x64 (conv2d block [conv2d, batchnorm2d, relu])

32x32x32 (conv2d block [conv2d, batchnorm2d, relu])

64x16x16 (conv2d block [conv2d, batchnorm2d, relu])

128x8x8 (conv2d block [conv2d, batchnorm2d, relu])

Flatten to 8192

2048 (linear block [linear, batchnorm1d, relu])

Split the 2048 dimension into mu and log variance for the parameters of the latent distribution

Latent mu size 256 (linear layer only with bias)

Latent logvar size 256 (linear layer only with bias)

In the middle here you can break out the BCE and KLD loss for the final loss term and use the standard reparam trick to sample from the latent distribution.

Decoder architecture an exact mirror

Input 256

2048 (linear block [linear, relu])

8192 (linear block [linear, batchnorm1d, relu])

reshape (128x8x8)

64x16x16 (conv2d transpose block [convtranspose2d, batchnorm2d, relu])

32x32x32 (conv2d transpose block [convtranspose2d, batchnorm2d, relu])

16x64x64 (conv2d transpose block [convtranspose2d, batchnorm2d, relu])

3x128x128 (conv2d transpose [convtranspose2d, sigmoid]

For weight initialization I used a normal distribution centered at zero with 0.02 set for the stddev. Optimizer: Adam with default parameters, if I were to do it over again I'd spend more time here understanding the learning dynamics. The dataset was about ~10,000 with a 70/20/10 split, batch size 64, over 120 epochs, with a learning schedule to reduce when the loss started to plateau. No crazy image augmentation just resizing and standards flips. I used the ANN package Annoy to do the NN search for prod, normalizing the embeddings and using the cosine similarity, ANN factor was 128 for num_trees.

MMD regularized VAE where the latents are drawn from a

TODO: Ranger optimizer might really help .. test

We can also use the transfer learning VAE tooling we previously built. We just need to create the convolutional encoder and pass it in... Note that we don't have a pre-trained option, so DON'T FREEZE!

Now just wrap that simple conv block architecture into a builder. And a meta-wrapper to let us call the conv_encoder and pre-trained options with the same function. (I'll also put the get_pretrained_parts function here now even though we won't use it till the next section, so that we can make the get_encoder_parts generic wrapper handle both properly.)

latent_dim = 128

# equalize KLDiv wrt errors per pixel
alpha = 3*IMG_SIZE*IMG_SIZE/latent_dim
alpha /= 20  # 5% retularizer

batchmean = True 
useL1 = False
hidden_dim = None
# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),  ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

metrics = default_AE_metrics(alpha,batchmean,useL1)

block = get_ae_DataBlock(aug=True)
batch_size = 64
dls = block.dataloaders(df, batch_size=batch_size)

arch='vanilla'

vae = AE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = AELoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,

{% include note.html content='The to_fp16() callbacks work but increasing the batch size doesn’t really speed things up.' %}

lr1,lr2=learn.lr_find()

mlr, gmlr = .5*(lr1+lr2), torch.tensor([lr1,lr2]).log().mean().exp().tolist()
lr1,lr2,mlr,gmlr

(0.006918309628963471,
 0.03981071710586548,
 0.023364513367414474,
 0.0165958721190691)

n_epoch = 100
learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)


# n_epoch = 10
# learn.fit_one_cycle(n_epoch,lr_max=lr1) #, lr_max= base_lr)

# learn.show_results()

learn.show_results()

prefix = f"AE-{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

# x_hat,latents = learn.model.cuda()(x)

# dummy_var1 = (z.var(dim=1).unsqueeze(-1).expand(z.size()) )
# dummy_var2 = (z.var(dim=0).unsqueeze(0).expand(z.size()) )

# dummy_var1[:10,0],dummy_var2[:10,0]

constructing VAE with Module Class Layers

For several of the decoder and "sampler" layers I might want to turn off the nonlinearity to give us more reasonable "gaussian" outputs to the Variational layer and the generated image which will is compared with the ImageNetStats batch-normalized image.

IMPORTANT VAE TIP!!! Make sure NOT to use batch normalization and non-linearity in the linear layers of the VAE. The normalization will affect the representation and the KLD constraints.

simple VAE class from VAE layer class components

once we have this we can do three things:

1. develop proper VAE loss functions (including KL Divergence constraint on latent variables)
2. create callbacks (and custom learner?) for training
3. extend to a beta-variational framework with aims at creating "disentangled" latent dimensions

Putting it all together gives us our VAE! Note that we'll pass in the "parts" of the encoder for ease of using pretrained (or not) architectures. The model name will correspond to the architecture of the encoder via name.

Note that the BVAE can simply inherit from the AE class we defined above. Really the only difference in the __init__ function is that a VAELayer which performs the reparameterization trick replaces the AElayer as self.bn

### TODO:  refactor the BVAE and AE to a single architecture... with a "sample" function ot 

class BVAE(AE):
    """
    simple VAE made with an encoder passed in, and some builder function for the Latent (VAE reparam trick) and decoder
    """
    def __init__(self,enc_parts,hidden_dim=None, latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE):
       
        """
        inputs:  
            enc_arch (pre-cut / pretrained)
            enc_dim
            latent_dim
            hidden_dim
            im_size,out_range
        """
        enc_arch,enc_feats,name = enc_parts

        # encoder
        #  arch,cut = xresnet18(pretrained=True),-4
        #  enc_arch = list(arch.children())[:cut]
        
        BASE = im_size//2**5
        enc_dim = enc_feats * BASE**2  # 2**(3*3) * (im_size//32)**2 #(output of resneet) #12800

        self.encoder = build_AE_encoder(enc_arch,enc_dim=enc_dim, hidden_dim=hidden_dim, im_size=im_size)

        in_dim = enc_dim if hidden_dim is None else hidden_dim

        # VAE Bottleneck
        self.bn = VAELayer(in_dim,latent_dim)     

        #decoder
        self.decoder = build_AE_decoder(hidden_dim=hidden_dim, latent_dim=latent_dim, im_size=im_size,out_range=out_range)

        store_attr('name,enc_dim, in_dim,hidden_dim,latent_dim,im_size,out_range') # do i need all these?


#     def decode(self, z):    
#         return self.decoder(z)

#     def encode(self, x):
#         h = self.encoder(x)
#         z, mu, logvar = self.bn(h) # reparam happens in the VAE layer
#         return z, mu, logvar

#     def forward(self, x):
#         #z, mu, logvar = self.encode(x)
#         #         h = self.encoder(x)
#         #         z, mu, logvar = self.bn(h) # reparam happens in the VAE layer
#         #         x_hat = self.decoder(z)
        
#         z,mu,logvar = self.encode(x)
#         x_hat = self.decode(z)
#         latents = torch.stack([mu,logvar],dim=-1)
        
#         return x_hat, latents # assume dims are [batch,latent_dim,concat_dim]

    
# # AE 
#    def decode(self, z):    
#         return self.decoder(z)
    
#     def encode(self, x):
#         h = self.encoder(x)
#         return self.bn(h)
            
#     def forward(self, x):
#         """
#         pass the "latents" out to keep the learn mechanics consistent... 
#         """
#         h = self.encoder(x)
#         z,logvar = self.bn(h)
#         x_hat = self.decoder(z)                    
#         latents = torch.stack([z,logvar] ,dim=-1)

#         return x_hat , latents

A nice wrapper for building the encoder parts will be handy.

Sweet, we've verified the arcitecture works, but we need to train it with a loss that constrains the variational layers with the KL Divergence. Otherwise the simple MSE will diverge.

VAE Loss functions classes

We have acouple examples to follow:

1. TabularData Vae (fastai v2 patterning) (@EtienneT)

~~pure PyTorch Vae which is directly related to our image dataset (@AntixK)~~

latent_dim = 128

# equalize KLDiv wrt errors per pixel
#  alpha = 3*IMG_SIZE*IMG_SIZE/latent_dim
alpha = 5
batchmean = True
useL1 = False
hidden_dim = None

# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
# start with KLweight on completely to enable sensible lr_find results
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]

metrics = default_VAE_metrics(alpha,batchmean,useL1)

block = get_ae_DataBlock(aug=True)


batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

# arch,cut = resnet18(pretrained=True),-4
# enc_arch = list(arch.children())[:cut]
# enc_dim = 512
# enc_parts = enc_arch,enc_dim,'resnet18'

# replaced by:
enc_parts = get_pretrained_parts(arch=resnet18)

rnet_vae = BVAE(enc_parts, hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)
    
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)
learn = Learner(dls, rnet_vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

# learn = learn.to_fp16()

#learn.show_training_loop()

learn.freeze()

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.0002511886414140463,
 0.0003311311302240938,
 0.00029115988581907004,
 0.00028840304003097117)

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler

learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )

<fastai.learner.Learner at 0x7f10bfeca3a0>

# the defaults are pretty good for now
n_epochs = 20

learn.fit_one_cycle(5,lr_max= lr1)#, lr_max= base_lr)
#learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
#learn.fit_flat_cos(n_epochs)
learn.show_results()

The vae with pretrained resnet encoder seems to train to a much better end-point if we keep the resnet frozen. Hence the commented out learn.unfreeze() below.

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()

<fastai.learner.Learner at 0x7f10bfeca3a0>

learn.freeze()

lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(3.6307807022240014e-06,
 3.0199516913853586e-05,
 1.6915148808038794e-05,
 1.0471291716385167e-05)

base_lr = 1e-5# gmlr #/= 2
epochs = 20

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
learn.fit_flat_cos(epochs,pct_start=.05)

learn.show_results()

prefix = f"Vae-{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

"Vanilla" convolutional beta-VAE

latent_dim = 128

# equalize KLDiv wrt errors per pixel
#  alpha = 3*IMG_SIZE*IMG_SIZE/latent_dim
alpha = 5
batchmean = True
useL1 = False

# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight': default_KL_anneal_in() })]

metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='vanilla'
vae = BVAE(get_encoder_parts(arch), hidden_dim=None,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.13182567358016967,
 0.17378008365631104,
 0.15280287861824035,
 0.15135611593723297)

n_epoch = 40
learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
#learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()

# n_epoch = 40
# learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e6, pct_start=0.3)


# learn.show_results()

learn.remove_cb(learn.cbs[-1])

# add new scheduler
learn.add_cb(ParamScheduler({'kl_weight': SchedNo(1.,1.) }) )

<fastai.learner.Learner at 0x7f10bf558d90>

x=2

n_epoch = 100
#learn.fit_flat_cos(n_epoch, lr=1e-4, div_final=1e5, pct_start=0.4)
learn.fit_flat_cos(n_epoch) 
#learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()

prefix = f"Vae-{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

n_epoch = 100
learn.fit_flat_cos(n_epoch, lr=5e-5,  pct_start=0.5)
#learn.fit_flat_cos(n_epoch) 
#learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()

prefix = f"AE-{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"{prefix}-2-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

# h = vae.enc.cuda()(x)
# z,mu,logvar = vae.cuda().reparam(h)
# x_hat = vae.dec.cuda()(z)
# x_hat2,latents = vae.cuda()(x)
# h.shape,z.shape,x_hat.shape, x_hat2.shape,latents.shape

MMD-VAE

replace latent regularization term in loss_fn (KLD) with MMD

We'll make an MMDVAE class to keep things declarative, but its really just an AE. i.e. a linear latent layer.

Vanilla MMD VAE

latent_dim = 128

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
alpha = 20

batchmean = True
useL1 = False
hidden_dim = None
metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)


batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)


arch='vanilla'
vae = MMDVAE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,

# x,y,z = learn.dls.one_batch()
# x_hat,latents = learn.model(x)

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

# WARNING... the loss is way way way too high. to start with... so the numbers a bogus... 
# 10-3 is a good place to start

n_epoch = 100
# learn.fit_flat_cos(n_epoch, lr=1e-2)
#learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e6, pct_start=0.1)
# n_epoch = 10
learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

Higher alpha = 20

latent_dim = 128

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
alpha = 30

batchmean = True
useL1 = False

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)


batch_size = 64
dls = block.dataloaders(df, batch_size=batch_size)

arch='vanilla'
vae = MMDVAE(get_encoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)

# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

# WARNING... the loss is way way way too high. to start with... so the numbers a bogus... 
# 10-3 is a good place to start

(0.33113112449646,
 0.12022644281387329,
 0.22567878365516664,
 0.19952622056007385)

n_epoch = 100
learn.fit_flat_cos(n_epoch)#, lr=lr1)
#learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e6, pct_start=0.1)
#learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-43-4c62d66164de> in <module>
      1 # warm things up...  low-ish learning rate + anealing kldiv retularizaiton into
      2 n_epoch = 100
----> 3 learn.fit_flat_cos(n_epoch)#, lr=lr1)
      4 #learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e6, pct_start=0.1)
      5 #learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/callback/schedule.py in fit_flat_cos(self, n_epoch, lr, div_final, pct_start, wd, cbs, reset_opt)
    133     lr = np.array([h['lr'] for h in self.opt.hypers])
    134     scheds = {'lr': combined_cos(pct_start, lr, lr, lr/div_final)}
--> 135     self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd)
    136 
    137 # Cell

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt)
    209             self.opt.set_hypers(lr=self.lr if lr is None else lr)
    210             self.n_epoch = n_epoch
--> 211             self._with_events(self._do_fit, 'fit', CancelFitException, self._end_cleanup)
    212 
    213     def _end_cleanup(self): self.dl,self.xb,self.yb,self.pred,self.loss = None,(None,),(None,),None,None

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    158 
    159     def _with_events(self, f, event_type, ex, final=noop):
--> 160         try: self(f'before_{event_type}');  f()
    161         except ex: self(f'after_cancel_{event_type}')
    162         self(f'after_{event_type}');  final()

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _do_fit(self)
    200         for epoch in range(self.n_epoch):
    201             self.epoch=epoch
--> 202             self._with_events(self._do_epoch, 'epoch', CancelEpochException)
    203 
    204     def fit(self, n_epoch, lr=None, wd=None, cbs=None, reset_opt=False):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    158 
    159     def _with_events(self, f, event_type, ex, final=noop):
--> 160         try: self(f'before_{event_type}');  f()
    161         except ex: self(f'after_cancel_{event_type}')
    162         self(f'after_{event_type}');  final()

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _do_epoch(self)
    194 
    195     def _do_epoch(self):
--> 196         self._do_epoch_train()
    197         self._do_epoch_validate()
    198 

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _do_epoch_train(self)
    186     def _do_epoch_train(self):
    187         self.dl = self.dls.train
--> 188         self._with_events(self.all_batches, 'train', CancelTrainException)
    189 
    190     def _do_epoch_validate(self, ds_idx=1, dl=None):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    158 
    159     def _with_events(self, f, event_type, ex, final=noop):
--> 160         try: self(f'before_{event_type}');  f()
    161         except ex: self(f'after_cancel_{event_type}')
    162         self(f'after_{event_type}');  final()

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in all_batches(self)
    164     def all_batches(self):
    165         self.n_iter = len(self.dl)
--> 166         for o in enumerate(self.dl): self.one_batch(*o)
    167 
    168     def _do_one_batch(self):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in one_batch(self, i, b)
    182         self.iter = i
    183         self._split(b)
--> 184         self._with_events(self._do_one_batch, 'batch', CancelBatchException)
    185 
    186     def _do_epoch_train(self):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    158 
    159     def _with_events(self, f, event_type, ex, final=noop):
--> 160         try: self(f'before_{event_type}');  f()
    161         except ex: self(f'after_cancel_{event_type}')
    162         self(f'after_{event_type}');  final()

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in __call__(self, event_name)
    139 
    140     def ordered_cbs(self, event): return [cb for cb in self.cbs.sorted('order') if hasattr(cb, event)]
--> 141     def __call__(self, event_name): L(event_name).map(self._call_one)
    142 
    143     def _call_one(self, event_name):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastcore/foundation.py in map(self, f, gen, *args, **kwargs)
    152     def range(cls, a, b=None, step=None): return cls(range_of(a, b=b, step=step))
    153 
--> 154     def map(self, f, *args, gen=False, **kwargs): return self._new(map_ex(self, f, *args, gen=gen, **kwargs))
    155     def argwhere(self, f, negate=False, **kwargs): return self._new(argwhere(self, f, negate, **kwargs))
    156     def filter(self, f=noop, negate=False, gen=False, **kwargs):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastcore/basics.py in map_ex(iterable, f, gen, *args, **kwargs)
    664     res = map(g, iterable)
    665     if gen: return res
--> 666     return list(res)
    667 
    668 # Cell

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastcore/basics.py in __call__(self, *args, **kwargs)
    649             if isinstance(v,_Arg): kwargs[k] = args.pop(v.i)
    650         fargs = [args[x.i] if isinstance(x, _Arg) else x for x in self.pargs] + args[self.maxi+1:]
--> 651         return self.func(*fargs, **kwargs)
    652 
    653 # Cell

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _call_one(self, event_name)
    143     def _call_one(self, event_name):
    144         if not hasattr(event, event_name): raise Exception(f'missing {event_name}')
--> 145         for cb in self.cbs.sorted('order'): cb(event_name)
    146 
    147     def _bn_bias_state(self, with_bias): return norm_bias_params(self.model, with_bias).map(self.opt.state)

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/callback/core.py in __call__(self, event_name)
     42                (self.run_valid and not getattr(self, 'training', False)))
     43         res = None
---> 44         if self.run and _run: res = getattr(self, event_name, noop)()
     45         if event_name=='after_fit': self.run=True #Reset self.run to True at each end of fit
     46         return res

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/callback/schedule.py in before_batch(self)
     84     def __init__(self, scheds): self.scheds = scheds
     85     def before_fit(self): self.hps = {p:[] for p in self.scheds.keys()}
---> 86     def before_batch(self): self._update_val(self.pct_train)
     87 
     88     def _update_val(self, pct):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/callback/schedule.py in _update_val(self, pct)
     87 
     88     def _update_val(self, pct):
---> 89         for n,f in self.scheds.items(): self.opt.set_hyper(n, f(pct))
     90 
     91     def after_batch(self):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/callback/schedule.py in _inner(pos)
     66     def _inner(pos):
     67         if int(pos) == 1: return scheds[-1](1.)
---> 68         idx = (pos >= pcts).nonzero().max()
     69         actual_pos = (pos-pcts[idx]) / (pcts[idx+1]-pcts[idx])
     70         return scheds[idx](actual_pos.item())

KeyboardInterrupt:

prefix = f"MMDVae-{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

ResNet Encoder MMDVAE

# # cut = model_meta[resnet50]['cut']
# # enc_arch = list(arch.children())[:cut]
# # nn.Sequential(*enc_arch).cuda()(x).shape
# arch=resnet18
# cut = model_meta[arch]['cut']
# arch = arch(pretrained=True)
# enc_arch = list(arch.children())[:cut]
# enc_dim = 512
# vae = MMDVAE(enc_arch,enc_dim=enc_dim, hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)

L2 error func

    
    
latent_dim = 128

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
alpha = 10

batchmean = True
useL1 = False
hidden_dim = None
metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)


batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)


arch=resnet18
vae = MMDVAE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,
    
learn.freeze()

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

learn.unfreeze()
lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'TMP'}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')

n_epoch = 5
learn.unfreeze()
learn.fit_flat_cos(n_epoch, lr=lr1, div_final=1e6, pct_start=0.7)
#learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e5, pct_start=0.5)
learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"unfrozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

    
    
latent_dim = 128

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
alpha = 20

batchmean = True
useL1 = False
hidden_dim = None
metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)


batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)


arch=resnet18
vae = MMDVAE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,
    
learn.freeze()

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
#learn.fit_flat_cos(n_epoch, lr=lr1, div_final=1e5, pct_start=0.5)
learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

learn.unfreeze()
lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

#n_epoch = 100
#learn.fit_flat_cos(n_epoch, lr=lr1, div_final=1e6, pct_start=0.05)
#learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e5, pct_start=0.5)
learn.fit_one_cycle(5) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"unfrozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

resnet arch no freeze

vanilla MMD VAE

# # cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
# #                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
# #                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

# alpha = 8
# batchmean = False

# metrics = [MMDMetric(batchmean=batchmean,alpha=alpha), 
#            L1MeanMetric(), 
#            MuMetric(), 
#            StdMetric(),
#            LogvarMetric(),
#            L2MeanMetric(),
#            WeightedKLDMetric(batchmean=batchmean,alpha=alpha),
#            KLWeightMetric(), 
#            MuSDMetric(), 
#            LogvarSDMetric(),
#            RawKLDMetric(batchmean=batchmean)
#           ]


# batch_size = 64
# dls = block.dataloaders(df, batch_size=batch_size)

    
latent_dim = 128

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
alpha = 10

batchmean = True
useL1 = False
hidden_dim = None
metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)


batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)


arch=resnet18
vae = MMDVAE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,
    
learn.unfreeze()

n_epoch = 200
learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e6, pct_start=0.1)

learn.show_results()

prefix = f"MMDVae-nofreeze{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

    
latent_dim = 128

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
alpha = 20

batchmean = True
useL1 = False
hidden_dim = None
metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)


batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)


arch=resnet18
vae = MMDVAE(get_encoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim, im_size=IMG_SIZE,out_range=OUT_RANGE)

# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=useL1)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split) #.to_fp16() #wd=config['wd'],opt_func=ranger,
    
learn.unfreeze()

n_epoch = 200
learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e6, pct_start=0.1)

learn.show_results()

prefix = f"MMDVae-nofreeze{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

vanilla-resnet MMD-VAE

ResBlocks replacing Conv2d in encoder AND decoder.

'Vanilla' encoder made with resnet blocks instead of conv blocks and Mish activation

useL1 = False

def get_conv_parts2(enc_type='vanilla',im_size=IMG_SIZE):
    """
    make a simple convolutional ladder encoder
    TODO:  make a switch on enc_type
        - vanilla, resnet or vanilla-res
        -TODO: change to 'convblock' and 'resblock'
    """
    n_blocks = 5
    BASE = im_size//2**5        
    nfs = [3]+[(2**i)*BASE for i in range(n_blocks)] 
    n = len(nfs)

    
    
    modules =  [ConvLayer(nfs[i],nfs[i+1],
                            ks=5,stride=2,padding=2) for i in range(n - 1)]       
    
#     modules =  [ResBlock(1, nfs[i],nfs[i+1], 
#                          stride=2, act_cls=Mish)  for i in range(n - 1)]       

    return modules,nfs[-1],'vanilla'


def get_encoder_parts2(enc_type='vanilla',im_size=IMG_SIZE):
    encoder_parts = get_conv_parts2(enc_type=enc_type,im_size=im_size) if isinstance(enc_type,str) else get_pretrained_parts(arch=enc_type)
    return encoder_parts # returns enc_arch,enc_dim,arch.__name__

- vanilla, resnet or vanilla-res
-TODO: change to 'convblock' and 'resblock'

128 latents, alpha=10

latent_dim = 128

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 10
# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
batchmean = True
useL1 = False
hidden_dim = None

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)

batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim,  im_size=IMG_SIZE,out_range=OUT_RANGE)
  
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.33113112449646,
 0.03981071710586548,
 0.18547092080116273,
 0.11481534689664841)

#learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'TMP'}-latent{latent_dim}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')

Path('models/frozen-MMDVae-TMP-latent128-resblock-alpha10_2021-03-24_10.15.31.pth')

128 latents, alpha=20

latent_dim = 128

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 20
# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
batchmean = True
useL1 = False
hidden_dim = None

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)

batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim,  im_size=IMG_SIZE,out_range=OUT_RANGE)
  
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.33113112449646,
 0.04786301031708717,
 0.18949706740677358,
 0.1258925348520279)

#learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'TMP'}-latent{latent_dim}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')

Path('models/frozen-MMDVae-TMP-latent128-resblock-alpha20_2021-03-24_11.36.20.pth')

64 latents, alpha=10

latent_dim = 64

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 10
# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
batchmean = True
useL1 = False
hidden_dim = None

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)

batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim,  im_size=IMG_SIZE,out_range=OUT_RANGE)
  
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.33113112449646,
 0.05754399299621582,
 0.1943375587463379,
 0.13803842663764954)

#learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'TMP'}-latent{latent_dim}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')

Path('models/frozen-MMDVae-TMP-latent64-resblock-alpha10_2021-03-24_12.57.06.pth')

64 latents, alpha = 20

latent_dim = 64

# cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
#                SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True),
#                ParamScheduler({'kl_weight': SchedNo(1.,1.) })]
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(),         
               ParamScheduler({'kl_weight': SchedNo(1.,1.) })]

alpha = 10
# note that alpha needs to be adjusted to scale MMD regularizer compared to error for batchmean=true
#.  e.g.  *= 3*IMG_SIZE**2/latent_dim
batchmean = True
useL1 = False
hidden_dim = None

metrics = default_MMEVAE_metrics(alpha,batchmean,useL1)

batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resblock'
vae = ResBlockAE(get_resblockencoder_parts(arch), hidden_dim=hidden_dim,latent_dim=latent_dim,  im_size=IMG_SIZE,out_range=OUT_RANGE)
  
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = MMDLoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.33113112449646,
 0.03981071710586548,
 0.18547092080116273,
 0.11481534689664841)

#learn.freeze()
n_epoch = 200
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch)#, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch,lr_max=gmlr) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'TMP'}-latent{latent_dim}"
filename = f"frozen-{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
#learn.export(f'{filename}.pkl')

Path('models/frozen-MMDVae-TMP-latent64-resblock-alpha10_2021-03-24_14.17.51.pth')

vanilla-resnet beta-VAE

class ResBlockBVAE(BVAE):
    """
    simple VAE with a _probably_ pretrained encoder 
    """

    def __init__(self,enc_parts,hidden_dim=None, latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE):
       
        """
        inputs:  
            enc_arch (pre-cut / pretrained)
            enc_dim
            latent_dim
            hidden_dim
            im_size,out_range
        """
        enc_arch,enc_feats,name = enc_parts

        # encoder
        #  arch,cut = xresnet18(pretrained=True),-4
        #  enc_arch = list(arch.children())[:cut]
        
        BASE = im_size//2**5
        enc_dim = enc_feats * BASE**2  # 2**(3*3) * (im_size//32)**2 #(output of resneet) #12800

        self.encoder = build_AE_encoder(enc_arch,enc_dim=enc_dim, hidden_dim=hidden_dim, im_size=im_size)

        in_dim = enc_dim if hidden_dim is None else hidden_dim

        # VAE Bottleneck
        self.bn = VAELayer(in_dim,latent_dim)     

        #decoder
        self.decoder = build_ResBlockAE_decoder(hidden_dim=hidden_dim, latent_dim=latent_dim, im_size=im_size,out_range=out_range)

        store_attr('name,enc_dim, in_dim,hidden_dim,latent_dim,im_size,out_range') # do i need all these?




#  THESE ARE INHERITED..
#     def decode(self, z):    
#         z = self.decoder(z)
#         return z
    
#     def reparam(self, h):
#         return self.bn(h)

#     def encode(self, x):
#         h = self.encoder(x)
#         z, mu, logvar = self.reparam(h)
#         return z, mu, logvar

#     def forward(self, x):
#         z, mu, logvar = self.encode(x)
#         x_hat = self.decode(z)
#         latents = torch.stack([mu,logvar],dim=-1)
#         return x_hat, latents # assume dims are [batch,latent_dim,concat_dim]

128 latents, alpha=5

latent_dim = 128
alpha = 5 # doubled because latent is half?
batchmean = True
useL1 = False

# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockBVAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.0001737800776027143,
 0.0012022644514217973,
 0.0006880222645122558,
 0.0004570883756969124)

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )

<fastai.learner.Learner at 0x7f2e8e9203d0>

# the defaults are pretty good for now
n_epochs = 10

#learn.fit_one_cycle(freeze_epochs1,lr_max= lr1)#, lr_max= base_lr)
#learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
#learn.fit_flat_cos(n_epochs) #, lr=1e-4,pct_start=0.5)
learn.fit_one_cycle(n_epochs)#, lr_max= base_lr)

learn.show_results()

This initial "burning in" of the KLD regularization is very unstable...

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()

<fastai.learner.Learner at 0x7f2e8e9203d0>

lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(6.309573450380412e-08,
 1.5848931980144698e-06,
 8.239944662591369e-07,
 3.162276129842212e-07)

base_lr = 1e-5# gmlr #/= 2
epochs = 100

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
#learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
learn.fit_flat_cos(epochs,div_final=1000)#,lr=1e-4)
learn.show_results()

prefix = f"BVae-{'2step10_100'}-latent{latent_dim}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

x = 1

128 latents, alpha=10

latent_dim = 128
alpha = 10 # doubled because latent is half?
batchmean = True
useL1 = False



# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockBVAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.00043651582673192023,
 0.002511886414140463,
 0.0014742011204361915,
 0.001047128695063293)

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )

<fastai.learner.Learner at 0x7f2e8e940af0>

# # the defaults are pretty good for now
n_epochs = 10

learn.fit_one_cycle(n_epochs)#, lr_max= base_lr)
# #learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
# learn.fit_flat_cos(n_epochs, lr=1e-4,pct_start=0.5)

# learn.show_results()

# # the defaults are pretty good for now
# n_epochs = 10

# learn.fit_one_cycle(10)#, lr_max= base_lr)
# #learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
# #learn.fit_flat_cos(n_epochs, lr=1e-4,pct_start=0.5)

# learn.show_results()

This initial "burning in" of the KLD regularization is very unstable...

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()

<fastai.learner.Learner at 0x7f2e8e940af0>

lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(6.918309954926372e-05,
 1.4454397387453355e-05,
 4.181874846835854e-05,
 3.16227633447852e-05)

epochs = 100

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
#learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
learn.fit_flat_cos(epochs,div_final = 1000)#,lr=1e-4)
learn.show_results()

learn.lr

0.001

# filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

# learn.save(filename)
# learn.export(f'{filename}.pkl')

# base_lr = 1e-5# gmlr #/= 2
# epochs = 50

# #learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
# #learn.fit_one_cycle(epochs, lr_max= 1e-3)
# #learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
# learn.fit_flat_cos(epochs)
# learn.show_results()

# filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

# learn.save(filename)
# learn.export(f'{filename}.pkl')

# filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"
# filename = "BVae-POST-1CYCLE10-latent128-resblock-alpha10_2021-03-24_21.33.02"
# learn.load(filename)
# #epochs = 5
# epochs = 10
# learn.fit_one_cycle(epochs, lr_max=.001)
# #learn.fit_flat_cos(epochs,lr=.0015,pct_start=.5,div_final=1000.0)
# #learn.fit_one_cycle(epochs,lr_max=5e-3,pct_start=0.5,div_final=100000) # gets down to ~4500 loss in 10
# learn.show_results()

prefix = f"BVae-{'2step10_100'}-latent{latent_dim}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

64 latents, alpha=5

latent_dim = 64
alpha = 5 # doubled because latent is half?
batchmean = True
useL1 = False


# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockBVAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.0002511886414140463,
 0.0030199517495930195,
 0.0016355701955035329,
 0.0008709632093086839)

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )

<fastai.learner.Learner at 0x7f2e8f2804f0>

n_epochs = 10

learn.fit_one_cycle(n_epochs)#,lr_max= lr1)#, lr_max= base_lr)
#learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
#learn.fit_flat_cos(n_epochs, lr=1e-4,pct_start=0.5)

learn.show_results()

This initial "burning in" of the KLD regularization is very unstable...

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()

<fastai.learner.Learner at 0x7f2e8f2804f0>

lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(5.754399353463669e-07,
 2.2908675418875646e-06,
 1.4331537386169657e-06,
 1.1481544106572983e-06)

base_lr = 1e-5# gmlr #/= 2
epochs = 100

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
#learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
#learn.fit_flat_cos(epochs,lr=1e-4)
learn.fit_flat_cos(epochs, div_final=1000.0)#,lr=1e-3)
learn.show_results()

prefix = f"BVae-{'2step10_100'}-latent{latent_dim}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

64 latents, alpha=10

latent_dim = 64
alpha = 10 # doubled because latent is half?
batchmean = True
useL1 = False


# SaveModelCallback(fname=datetime.now().strftime('%Y-%m-%d %Hh%M.%S'), every_epoch=True), 
cbs = [AnnealedLossCallback(),TerminateOnNaNCallback(), ParamScheduler({'kl_weight':  SchedNo(1.,1.) })]


metrics = default_VAE_metrics(alpha,batchmean,useL1)


block = get_ae_DataBlock(aug=True)
batch_size = 128
dls = block.dataloaders(df, batch_size=batch_size)

arch='resnblock'
vae = ResBlockBVAE(get_resblockencoder_parts(arch), hidden_dim=2048,latent_dim=128, im_size=IMG_SIZE,out_range=OUT_RANGE)
                   
# let beta be calculated by : 3*im_size*im_size/latent_dim
loss_func = BVAELoss(batchmean=batchmean,alpha=alpha,useL1=False)

learn = Learner(dls, vae, cbs=cbs,loss_func=loss_func, metrics=metrics,splitter=AE_split)#.to_fp16() #wd=config['wd'],opt_func=ranger,

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(0.00043651582673192023,
 0.00363078061491251,
 0.002033648220822215,
 0.0012589251855388284)

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight': default_KL_anneal_in()} ) )

<fastai.learner.Learner at 0x7f2e8f2327f0>

# the defaults are pretty good for now
n_epochs = 10

learn.fit_one_cycle(n_epochs)#,lr_max= lr1)#, lr_max= base_lr)
#learn.fit_flat_cos(n_epochs, lr=lr1, pct_start=0.5)
#learn.fit_flat_cos(n_epochs, lr=1e-4,pct_start=0.5)

learn.show_results()

This initial "burning in" of the KLD regularization is very unstable...

learn.remove_cb(learn.cbs[-1])
# add new constant scheduler
learn.add_cb(ParamScheduler({'kl_weight':  SchedNo(1.,1.) }) )

#learn.unfreeze()

<fastai.learner.Learner at 0x7f2e8f2327f0>

lr1,lr2=learn.lr_find()

mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

(6.309573450380412e-08,
 1.9054607491852948e-06,
 9.842782418445494e-07,
 3.467365559117752e-07)

base_lr = 1e-5# gmlr #/= 2
epochs = 100

#learn.fit_one_cycle(epochs, slice(base_lr/lr_mult, base_lr), pct_start=pct_start, div=div)
#learn.fit_one_cycle(epochs, lr_max= 1e-3)
#learn.fit_flat_cos(epochs,lr=lr1,pct_start=.05)
learn.fit_flat_cos(epochs,div_final=1000.)
learn.show_results()

prefix = f"BVae-{'2step10_100'}-latent{latent_dim}"
filename = f"{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

next steps.

visualize activations in encoder decoder
improve training with progressive learning rates
G

export / provenance

CODE graveyard

class LatentTuple(fastuple):
    "Basic type for tuple of tensor (vectors)"
    _show_args = dict(s=10, marker='.', c='r')
    @classmethod
    def create(cls, ts): 
        if isinstance(ts,tuple):
            mu,logvar = ts
        elif ts is None:
            mu,logvar = None,None
        else:
            mu = None
            logvar = None
            
        if mu is None: mu = torch.empty(0)
        elif not isinstance(mu, Tensor): Tensor(mu) 
        
        if logvar is None: logvar = torch.empty(0)
        elif not isinstance(logvar,Tensor): Tensor(logvar)
            
        return cls( (mu,logvar) ) 
        
    def show(self, ctx=None, **kwargs): 
        mu,logvar = self
        if not isinstance(mu, Tensor) or not isinstance(logvar,Tensor): return ctx

        title_str = f"mu-> {mu.mean():e}, {mu.std():e}  logvar->{logvar.mean():e}, {logvar.std():e}"
    
        if 'figsize' in kwargs: del kwargs['figsize']
        if 'title' in kwargs: kwargs['title']=title_str
        if ctx is None:
            _,axs = plt.subplots(1,2, figsize=(12,6))
            x=torch.linspace(0,1,mu[0].shape[0])
            axs[0].scatter(x, mu[:], **{**self._show_args, **kwargs})
            axs[1].scatter(x, logvar[:], **{**self._show_args, **kwargs})
            ctx = axs[1]
        
        ctx.scatter(mu[:], logvar[:], **{**self._show_args, **kwargs})
        return ctx
    
# could we do a typedispatch to manage the transforms...?
def VAETargetTupleBlock(): 
    return TransformBlock(type_tfms=VAETargetTuple.create, batch_tfms=IntToFloatTensor)

def LatentTupleBlock(): 
    return TransformBlock(type_tfms=LatentTuple.create, batch_tfms=noop)
    

# class TensorPoint(TensorBase):
#     "Basic type for points in an image"
#     _show_args = dict(s=10, marker='.', c='r')

#     @classmethod
#     def create(cls, t, img_size=None)->None:
#         "Convert an array or a list of points `t` to a `Tensor`"
#         return cls(tensor(t).view(-1, 2).float(), img_size=img_size)

#     def show(self, ctx=None, **kwargs):
#         if 'figsize' in kwargs: del kwargs['figsize']
#         x = self.view(-1,2)
#         ctx.scatter(x[:, 0], x[:, 1], **{**self._show_args, **kwargs})
#         return ctx

latent_dim = 128
dropout = .2
im_size = IMG_SIZE
n_blocks = 5        
nfs = [3] + [2**i*n_blocks for i in range(n_blocks+1)] 
nfs.reverse()
# decoder = nn.Sequential(
#             nn.Linear(latent_size, 16),
#             UnFlatten(4),
#             ResBlock(1, 3, 4, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, 4, 8, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, 8, 16, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             nn.Conv2d(16, 1, 3),
#             nn.Dropout2d(dropout),
#             #nn.AdaptiveAvgPool2d((3,im_size, im_size)) 
# )





n_blocks = 5
hidden_dim = 2048
out_range = [-1,1]
tst = nn.Sequential(
            nn.Linear(latent_dim,hidden_dim), #nn.Linear(latent_dim, 16)
            nn.Linear(hidden_dim,im_size*n_blocks*n_blocks), #nn.Linear(latent_dim, 16)
            ResizeBatch(im_size,n_blocks,n_blocks),#UnFlatten(n_blocks), #4
            ResBlock(1, nfs[0], nfs[1], act_cls=Mish), #ResBlock(1, 1, 4, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[1], nfs[2], act_cls=Mish), #RResBlock(1, 4, 8, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[2], nfs[3], act_cls=Mish), #ResBlock(1, 8, 16, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[3], nfs[4], act_cls=Mish), #nn.Conv2d(16, 1, 3),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[4], nfs[5], act_cls=Mish), #nn.Conv2d(16, 1, 3),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[5], nfs[6], act_cls=Mish), #nn.Conv2d(16, 1, 3),
            #nn.Dropout2d(dropout),
            #nn.Upsample(scale_factor=2),            #
            #nn.AdaptiveAvgPool2d((3,im_size, im_size)),
            SigmoidRange(*out_range), #nn.Sigmoid()
)
tst,nfs

inp=    torch.randn((32,latent_dim))
#last_size = model_sizes(tst ) #[-1][1]
#num_features_model(tst)
#last_size
#nfs
tst(inp).shape,nfs
#last_size

z_dim = 100
enc = nn.Sequential(
    ResBlock(1, 1, 16, act_cls=nn.ReLU, norm_type=None),
    nn.MaxPool2d(2, 2),
    ResBlock(1, 16, 4, act_cls=nn.ReLU, norm_type=None),
    nn.MaxPool2d(2, 2),
    Flatten()
)
# torch.Size([32, 1, 28, 28])
# torch.Size([32, 16, 28, 28])
# torch.Size([32, 16, 14, 14])
# torch.Size([32, 4, 14, 14])
# torch.Size([32, 4, 7, 7])
# torch.Size([32, 196])

latent_size = 100

enc = nn.Sequential(
            ResBlock(1, 3, 5, stride=2, act_cls=Mish),#  1->3
            ResBlock(1, 5, 5, stride=2, act_cls=Mish),
            ResBlock(1, 5, 1, stride=2, act_cls=Mish),
            Flatten(),
            nn.Linear(400, latent_size) # 16->400
        )
#  torch.Size([32, 1, 28, 28])
# torch.Size([32, 5, 14, 14])
# torch.Size([32, 5, 7, 7])
# torch.Size([32, 1, 4, 4])
# torch.Size([32, 16])
# torch.Size([32, 4])       
inp=    torch.randn((32,3,160,160))
for ii in range(0,8):
    print(enc[:ii](inp).shape)

z = enc(inp)

dropout=0
dec = nn.Sequential(
            nn.Linear(latent_size, 16),
            UnFlatten(4),
            ResBlock(1, 1, 4, act_cls=Mish), #4->5
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, 4, 8, act_cls=Mish),
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, 8, 16, act_cls=Mish),
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(16, 3, 3), #1->3
            #nn.Dropout2d(dropout),
            nn.AdaptiveAvgPool2d((28, 28)),
            nn.Sigmoid()
        )

# torch.Size([32, 4])
# torch.Size([32, 16])
# torch.Size([32, 1, 4, 4])
# torch.Size([32, 4, 4, 4])
# torch.Size([32, 4, 8, 8])
# torch.Size([32, 8, 8, 8])
# torch.Size([32, 8, 16, 16])
# torch.Size([32, 16, 16, 16])
# torch.Size([32, 16, 32, 32])
# torch.Size([32, 1, 30, 30])
# torch.Size([32, 1, 28, 28])


for ii in range(0,12):
    print(dec[:ii](z).shape)

n_blocks = 5
BASE = im_size//2**5        
nfs = [3]+[(2**i)*BASE for i in range(n_blocks)] 
n = len(nfs)
hidden_dim = 2048

BASE = im_size//2**5
        # encoder
in_dim = nfs[-1] * BASE**2 
        
modules =  [ResBlock(1, nfs[i],nfs[i+1], 
                          stride=2, act_cls=Mish)  for i in range(n - 1)]    
# enc =  nn.Sequential(
#                 ConvLayer(nfs[0],nfs[1],ks=5,stride=2,padding=2),
#                 ConvLayer(nfs[1],nfs[2],ks=5,stride=2,padding=2),
#                 ConvLayer(nfs[2],nfs[3],ks=5,stride=2,padding=2),
#                 ConvLayer(nfs[3],nfs[4],ks=5,stride=2,padding=2),
#                 ConvLayer(nfs[4],nfs[5],ks=5,stride=2,padding=2),
#                 Flatten(),
#                 LinBnDrop(in_dim,hidden_dim,bn=True,p=0.0,act=nn.ReLU(),lin_first=True)
#             )

enc =  nn.Sequential(*modules,
                Flatten(),
                LinBnDrop(in_dim,hidden_dim,bn=True,p=0.0,act=nn.ReLU(),lin_first=True)
            )

nfs.reverse()

print(nfs)
#last_size = model_sizes(enc, size=(28,28))[-1][1]



    
encoder = nn.Sequential(enc, nn.Linear(hidden_dim, z_dim))
        
decoder = nn.Sequential(
            nn.Linear(z_dim, im_size*n_blocks*n_blocks),
            ResizeBatch(im_size,n_blocks,n_blocks),#UnFlatten(n_blocks), #4
            ResBlock(1, nfs[0], nfs[1], ks=1, act_cls=nn.ReLU, norm_type=None),
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[1], nfs[2], act_cls=nn.ReLU, norm_type=None),
            #nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(nfs[2], 3, 3, padding=1),
            #nn.Dropout2d(dropout),
            nn.Sigmoid()
        )
#last_size

#             nn.Linear(latent_dim,hidden_dim), #nn.Linear(latent_dim, 16)
#             nn.Linear(hidden_dim,im_size*n_blocks*n_blocks), #nn.Linear(latent_dim, 16)
#             ResizeBatch(im_size,n_blocks,n_blocks),#UnFlatten(n_blocks), #4
#             ResBlock(1, nfs[0], nfs[1], act_cls=Mish), #ResBlock(1, 1, 4, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[1], nfs[2], act_cls=Mish), #RResBlock(1, 4, 8, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[2], nfs[3], act_cls=Mish), #ResBlock(1, 8, 16, act_cls=Mish),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[3], nfs[4], act_cls=Mish), #nn.Conv2d(16, 1, 3),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[4], nfs[5], act_cls=Mish), #nn.Conv2d(16, 1, 3),
#             nn.Dropout2d(dropout),
#             nn.Upsample(scale_factor=2),
#             ResBlock(1, nfs[5], nfs[6], act_cls=Mish), #nn.Conv2d(16, 1, 3),

            

inp=    torch.randn((32,3,160,160))

#encoder[:1](inp).shape
for ii in range(0,10):
    print(enc[:ii](inp).shape)

z = encoder(inp)
for ii in range(0,14):
    print(decoder[:ii](z).shape)

class UnFlatten(Module):
    def __init__(self, size=7):
        self.size = size
    def forward(self, input):
        return input.view(input.size(0), -1, self.size, self.size)

class MMD_VAE(Module):
    def __init__(self, latent_size):        
        self.encoder = nn.Sequential(
            ResBlock(1, 1, 5, stride=2, act_cls=Mish),
            ResBlock(1, 5, 5, stride=2, act_cls=Mish),
            ResBlock(1, 5, 1, stride=2, act_cls=Mish),
            Flatten(),
            nn.Linear(16, latent_size)
        )
        
        dropout=0
        
        self.decoder = nn.Sequential(
            nn.Linear(latent_size, 16),
            UnFlatten(4),
            ResBlock(1, 1, 4, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, 4, 8, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, 8, 16, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            nn.Conv2d(16, 1, 3),
            nn.Dropout2d(dropout),
            nn.AdaptiveAvgPool2d((28, 28)),
            nn.Sigmoid()
        )
        
    
    def forward(self, X):
        latent = self.encoder(X)
        return self.decoder(latent), latent

    
#decoder
n_blocks = 5        
nfs = [3] + [2**i*n_blocks for i in range(n_blocks+1)] 
nfs.reverse()
n = len(nfs)


tst = nn.Sequential(
            nn.Linear(latent_dim,hidden_dim, #nn.Linear(latent_dim, 16)
            nn.Linear(hidden_dim,im_size*n_blocks*n_blocks) #nn.Linear(latent_dim, 16)
            UnFlatten(n_blocks), #4
            ResBlock(1, nfs[0], nfs[1], act_cls=Mish), #ResBlock(1, 1, 4, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[1], nfs[2], act_cls=Mish), #RResBlock(1, 4, 8, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[2], nfs[3], act_cls=Mish), #ResBlock(1, 8, 16, act_cls=Mish),
            nn.Dropout2d(dropout),
            nn.Upsample(scale_factor=2),
            ResBlock(1, nfs[3], nfs[4], act_cls=Mish), #nn.Conv2d(16, 1, 3),
            nn.Dropout2d(dropout),
            nn.AdaptiveAvgPool2d((3,im_size, im_size)),
            SigmoidRange(*out_range)#nn.Sigmoid()

    
                                         *modules,
                                      ConvLayer(nfs[-2],nfs[-1],
                                                ks=1,padding=0, norm_type=None, #act_cls=nn.Sigmoid) )
                                                act_cls=partial(SigmoidRange, *out_range)))

lr1,lr2=learn.lr_find()
mlr = .5*(lr1+lr2)
#geometric mean
gmlr = torch.tensor([lr1,lr2]).log().mean().exp().tolist()

lr1,lr2,mlr,gmlr

n_epoch = 10
#learn.fit_flat_cos(n_epoch) #, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch, lr=lr1, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()

n_epoch = 40
#learn.unfreeze()
#learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e6, pct_start=0.2)
learn.fit_flat_cos(n_epoch, lr=lr1, div_final=1e6, pct_start=0.05)

#learn.fit_flat_cos(n_epoch, lr=1e-3, div_final=1e5, pct_start=0.5)
#learn.fit_one_cycle(n_epoch) #, lr_max= base_lr)

learn.show_results()

prefix = f"MMDVae-{'bmean' if batchmean else 'mean'}{'l1' if useL1 else 'l2'}"
filename = f"frozen{prefix}-{learn.model.name}-alpha{alpha:d}_{datetime.now().strftime('%Y-%m-%d_%H.%M.%S')}"

learn.save(filename)
learn.export(f'{filename}.pkl')

x=1
x

def create_encoder(nfs,ks,conv=nn.Conv2d,bn=nn.BatchNorm2d,act_fn = nn.ReLU):
    """
    constructor for generic convolutional encoder 
    """
    n = len(nfs)
    conv_layers = [nn.Sequential(ConvBnRelu(nfs[i],nfs[i+1],kernel_size=ks[i],
                                            conv = conv,bn=bn,act_fn=act_fn, padding = ks[i] //2 ),
                                 Downsample(channels=nfs[i+1],filt_size=3,stride=2))
                                   for i in range(n-1)]        
    convs = nn.Sequential(*conv_layers)
    return convs

def create_encoder_denseblock(n_dense,c_start):
    """
    constructor for resnet with dense blocks  (?) 

    n_dense": 3,
    "c_start": 4
    """
    first_layer = nn.Sequential(ConvBnRelu(3,c_start,kernel_size=3,padding = 1),
                                ResBlock(c_start),
                                Downsample(channels=4,filt_size=3,stride=2))
    
    layers = [first_layer] + [
        nn.Sequential(
            DenseBlock(c_start * (2**c)),
            Downsample(channels=c_start * (2**(c+1)),filt_size=3,stride=2)) for c in range(n_dense)
    ]
    
    model = nn.Sequential(*layers)
    
    return model

def create_decoder(nfs, ks, size, conv=nn.Conv2d, bn=nn.BatchNorm2d, act_fn=nn.ReLU):
    """
    CURR VALUES:
    "nfs":[66,3*32,3*16,3*8,3*4,3*2,3,1,3],
    "ks": [ 3, 1, 3,1,3,1,3,1],   
    "size": IMG_SIZE 
    """
    n = len(nfs)
    
    # We add two channels to the first layer to include x and y channels
    first_layer = ConvBnRelu(nfs[0], #input size 
                             nfs[1], # output size
                             conv = PointwiseConv,
                             bn=bn,
                             act_fn=act_fn)

    conv_layers = [first_layer] + [ConvBnRelu(nfs[i],nfs[i+1],kernel_size=ks[i-1],
                                              padding = ks[i-1] // 2,conv = conv,bn=bn,act_fn=act_fn)
                                   for i in range(1,n - 1)]        
    dec_convs = nn.Sequential(*conv_layers)
    
    dec = nn.Sequential(SpatialDecoder2D(size),dec_convs)
    #SigmoidRange(*y_range)
    return dec

def decoder_simple(y_range=OUT_RANGE, n_out=3):
    return nn.Sequential(#UpsampleBlock(64),
                         UpsampleBlock(32),
                         nn.Conv2d(16, n_out, 1),
                         SigmoidRange(*y_range)
                        )

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	l1_b_mean	weighted_kld	l1_latent_reg	mu_sd	logvar_sd	time
0	128097.382812	134643.968750	39930.207031	-0.002537	0.040510	-6.754639	52492.644531	11061.582031	157.764160	0.062668	0.851084	00:17
1	73394.687500	62053.113281	24703.388672	0.000699	0.041367	-7.137185	39147.574219	11797.055664	62.264362	0.052086	1.219310	00:17
2	44016.871094	37138.363281	17046.394531	-0.000633	0.028182	-8.103985	29868.207031	13646.053711	33.511909	0.034439	1.362805	00:17
3	36447.683594	26723.453125	12383.161133	-0.000669	0.021824	-8.450100	22675.261719	14307.659180	23.908924	0.026107	1.280745	00:17
4	26018.755859	25212.011719	11840.586914	-0.000326	0.018731	-8.512531	19481.261719	14426.326172	22.302107	0.022185	1.171001	00:17
5	18737.417969	16063.942383	10955.346680	-0.000264	0.007900	-10.935346	17432.310547	19076.392578	8.516870	0.009517	1.381128	00:16
6	14920.210938	12629.469727	9424.293945	-0.000146	0.004278	-12.010656	15719.267578	21140.630859	5.342623	0.005707	1.527276	00:17
7	11940.507812	11961.258789	9256.583984	-0.000045	0.003518	-12.485510	14622.348633	22052.294922	4.508440	0.004661	1.524205	00:17
8	9525.660156	9208.806641	8052.059082	-0.000059	0.000692	-15.689737	13443.873047	28204.300781	1.927498	0.001370	1.629675	00:17
9	8524.423828	8587.276367	7683.373047	-0.000030	0.000349	-17.240650	12673.233398	31182.044922	1.505545	0.001042	1.855216	00:17
10	7978.969727	8156.777832	7329.306152	-0.000041	0.000315	-17.529308	12379.001953	31736.277344	1.378294	0.000904	1.938142	00:17
11	7987.073730	8534.000000	7489.396973	-0.000064	0.000385	-16.788462	12726.589844	30313.847656	1.740603	0.000978	1.695187	00:17
12	9005.134766	9590.705078	8276.330078	-0.000061	0.000551	-15.967548	13355.495117	28737.695312	2.189787	0.001290	1.612179	00:17
13	8016.973145	7898.334961	6934.157715	-0.000030	0.000370	-17.060190	12486.059570	30835.570312	1.607824	0.000989	1.854557	00:17
14	7528.673340	7442.922852	6461.455566	-0.000056	0.000393	-17.020239	11723.820312	30758.853516	1.635872	0.001070	1.946110	00:17
15	7334.169434	7185.149902	6285.567383	-0.000061	0.000342	-17.407375	11293.278320	31502.169922	1.499643	0.000948	1.990847	00:17
16	7161.321289	8549.125000	7613.462891	-0.000056	0.000357	-17.289757	12280.255859	31276.330078	1.559443	0.000932	1.987590	00:17
17	6871.318359	7131.771973	6242.437988	-0.000061	0.000332	-17.495859	11157.704102	31672.052734	1.481654	0.000920	2.034050	00:17
18	6947.179688	7371.867676	6305.343750	-0.000041	0.000389	-17.025923	11395.782227	30769.773438	1.778753	0.001003	1.929911	00:16
19	6544.943359	6460.646484	5733.094238	-0.000029	0.000286	-17.784206	10930.840820	32225.683594	1.213297	0.000758	2.020970	00:17
20	6537.779785	6866.859375	6052.525391	-0.000050	0.000295	-17.742332	10635.847656	32145.277344	1.357130	0.000784	2.057254	00:17
21	6465.080566	7321.693359	6523.076660	-0.000018	0.000288	-17.729782	11322.185547	32121.185547	1.329640	0.000756	2.001485	00:17
22	6472.735352	7526.670898	6691.609863	-0.000047	0.000330	-17.384459	11440.496094	31458.166016	1.391719	0.000810	1.947353	00:17
23	6488.197266	6530.095703	5662.607910	-0.000055	0.000325	-17.257601	10479.273438	31214.591797	1.446300	0.000786	1.841633	00:16
24	6255.183105	6429.492676	5674.761230	-0.000039	0.000286	-17.691570	10879.472656	32047.814453	1.258619	0.000708	1.978504	00:16
25	6089.413574	6213.600586	5482.145020	-0.000024	0.000281	-17.745461	10540.056641	32151.279297	1.220139	0.000708	1.983972	00:16
26	6418.912109	6311.363770	5572.911621	-0.000042	0.000287	-17.693209	10603.917969	32050.957031	1.230600	0.000727	1.984645	00:16
27	6196.784180	7820.942383	6796.470703	-0.000036	0.000317	-17.379375	11569.213867	31448.402344	1.706512	0.000842	1.915830	00:17
28	5916.974121	6442.241211	5748.355469	-0.000009	0.000257	-17.899462	10623.761719	32446.964844	1.156738	0.000650	1.971309	00:17
29	5935.304199	6784.444824	6001.813477	-0.000019	0.000276	-17.649734	10969.282227	31967.494141	1.304255	0.000669	1.894531	00:17
30	5879.327637	6364.133789	5525.782715	-0.000034	0.000316	-17.265694	10667.404297	31230.134766	1.398126	0.000744	1.783980	00:17
31	5904.208496	7091.926758	6408.140625	-0.000036	0.000287	-17.602674	10962.058594	31877.136719	1.138066	0.000690	1.921411	00:17
32	5949.079590	5831.209473	5152.445801	-0.000027	0.000264	-17.826963	10175.602539	32307.773438	1.131999	0.000649	1.976811	00:17
33	5741.439453	6124.986328	5409.072266	-0.000059	0.000286	-17.419025	10587.176758	31524.529297	1.193553	0.000686	1.789507	00:17
34	5757.232910	6776.560547	6049.399902	-0.000009	0.000270	-17.731647	10466.838867	32124.767578	1.211689	0.000666	1.940296	00:17
35	5639.804199	6053.663086	5359.206543	-0.000016	0.000240	-17.883928	10484.635742	32417.144531	1.157899	0.000592	1.885049	00:17
36	5752.002930	6832.224609	5999.978027	-0.000034	0.000285	-17.599562	10811.944336	31871.164062	1.386604	0.000719	1.926477	00:17
37	5627.188965	6259.631836	5549.889160	-0.000036	0.000281	-17.626829	10295.064453	31923.509766	1.183183	0.000663	1.920973	00:16
38	5616.164062	6181.266602	5443.573242	-0.000012	0.000255	-17.853054	9777.254883	32357.859375	1.229441	0.000630	1.969903	00:17
39	5505.995117	5579.451660	4870.064941	-0.000041	0.000255	-17.840361	9575.471680	32333.494141	1.182780	0.000614	1.957818	00:17
40	5411.520996	5808.485840	5178.082031	-0.000034	0.000241	-17.952686	10319.687500	32549.154297	1.050943	0.000572	1.960310	00:17
41	5427.710449	6076.456055	5418.766113	-0.000028	0.000251	-17.880894	11103.524414	32411.316406	1.096607	0.000596	1.974334	00:16
42	5381.585938	5839.857910	5207.262207	-0.000054	0.000234	-18.122000	9508.860352	32874.246094	1.054815	0.000566	2.060699	00:16
43	5333.306641	5924.423340	5315.966797	-0.000019	0.000212	-18.248751	10388.512695	33117.601562	1.014678	0.000509	2.017952	00:16
44	5383.250977	5644.471191	5018.856445	-0.000028	0.000232	-18.188360	9915.159180	33001.648438	1.043148	0.000570	2.120071	00:17
45	5311.410156	6503.749023	5849.697754	-0.000008	0.000270	-17.733797	9816.430664	32128.888672	1.090404	0.000583	1.940485	00:17
46	5151.930664	5536.779297	4923.996094	-0.000034	0.000206	-18.408070	9715.844727	33423.496094	1.021848	0.000510	2.108021	00:17
47	5297.787598	8156.237793	7548.040039	-0.000032	0.000252	-17.874067	12324.596680	32398.205078	1.010399	0.000542	1.980715	00:17
48	5227.941895	6154.923340	5534.933594	-0.000034	0.000229	-18.151596	10465.467773	32931.062500	1.033747	0.000551	2.081376	00:17
49	5124.862793	5773.290039	5180.436035	-0.000047	0.000220	-18.319500	9963.057617	33253.433594	0.988342	0.000532	2.151639	00:17
50	5075.948730	5529.555664	4981.696289	-0.000038	0.000211	-18.389603	9868.174805	33388.039062	0.913246	0.000501	2.128523	00:17
51	5010.145020	5811.954590	5283.138184	-0.000033	0.000200	-18.544312	10407.128906	33685.078125	0.881936	0.000485	2.172494	00:17
52	5264.053223	5603.073242	4987.989258	-0.000028	0.000219	-18.299126	10349.041016	33214.324219	1.025546	0.000526	2.123271	00:17
53	5113.655273	5667.930176	5039.185547	-0.000020	0.000223	-18.354670	9887.832031	33320.960938	1.048260	0.000546	2.195888	00:17
54	4991.845703	5516.799316	4957.778320	-0.000030	0.000205	-18.575159	9202.385742	33744.312500	0.932313	0.000507	2.231039	00:17
55	4986.747559	5663.084473	5095.498535	-0.000023	0.000190	-18.752609	10062.741211	34085.007812	0.945949	0.000471	2.264451	00:17
56	4922.537598	5784.844727	5246.645996	-0.000019	0.000188	-18.768650	10356.762695	34115.808594	0.897278	0.000458	2.259478	00:17
57	4966.143066	5423.006348	4859.388184	-0.000044	0.000201	-18.657448	9780.310547	33902.296875	0.939638	0.000493	2.286342	00:17
58	4915.844727	5541.731934	5018.822266	-0.000024	0.000187	-18.796347	9948.219727	34168.980469	0.871074	0.000456	2.281477	00:16
59	4963.224121	5233.171387	4640.420898	-0.000012	0.000205	-18.644356	9226.224609	33877.160156	0.988391	0.000507	2.296546	00:17
60	4965.743164	5909.845215	5292.252930	-0.000063	0.000220	-18.431940	9402.412109	33469.324219	1.029665	0.000536	2.229753	00:17
61	4932.851074	5525.992188	4933.409180	-0.000023	0.000204	-18.614147	9687.130859	33819.164062	0.987427	0.000498	2.265586	00:16
62	4883.003906	5925.290039	5322.096191	-0.000001	0.000204	-18.649969	10423.093750	33887.941406	1.004905	0.000506	2.280256	00:16
63	4827.962891	4992.815430	4469.872070	-0.000008	0.000189	-18.816412	8994.766602	34207.511719	0.872022	0.000456	2.319862	00:17
64	4792.510254	5500.577637	4993.454102	-0.000022	0.000172	-19.014614	10405.764648	34588.066406	0.844686	0.000428	2.323791	00:17
65	4826.964844	5292.974609	4721.527832	-0.000006	0.000193	-18.749868	9532.593750	34079.746094	0.952607	0.000475	2.298978	00:17
66	4800.906250	6924.175781	6351.299316	-0.000002	0.000189	-18.853163	10447.964844	34278.070312	0.953996	0.000480	2.346651	00:17
67	4817.762207	5239.083984	4670.365723	-0.000028	0.000183	-18.908623	9117.895508	34384.554688	0.948023	0.000469	2.337974	00:17
68	5745.058105	20536.539062	19835.625000	-0.000041	0.000218	-18.484266	20654.943359	33569.796875	1.156871	0.000569	2.277551	00:17
69	4962.108398	6923.226074	6358.041504	-0.000052	0.000187	-18.854486	11462.657227	34280.617188	0.941946	0.000478	2.328717	00:16
70	4821.537109	5796.459961	5237.106934	-0.000021	0.000195	-18.854546	10156.842773	34280.722656	0.932205	0.000491	2.396297	00:17
71	4764.200195	5999.174316	5476.904297	-0.000014	0.000181	-18.982876	9821.648438	34527.121094	0.870503	0.000448	2.388810	00:17
72	4847.364746	5096.290039	4558.476074	-0.000022	0.000193	-18.798292	9261.302734	34172.726562	0.896708	0.000468	2.339805	00:17
73	4801.708008	5929.185547	5332.706543	-0.000024	0.000196	-18.833414	9817.650391	34240.152344	0.994633	0.000499	2.385950	00:17
74	4692.241211	5155.827148	4610.053223	-0.000036	0.000182	-19.013065	8948.180664	34585.089844	0.909662	0.000464	2.409934	00:17
75	4722.820312	5203.173340	4668.738281	-0.000031	0.000188	-18.913076	9335.527344	34393.105469	0.890604	0.000460	2.394308	00:17
76	4729.158691	6752.458496	6183.010742	-0.000011	0.000182	-19.045420	10801.014648	34647.210938	0.949307	0.000467	2.442510	00:17
77	4711.308105	5911.178711	5391.027344	-0.000003	0.000178	-19.036831	9731.122070	34630.710938	0.867483	0.000447	2.402573	00:17
78	4647.219238	5577.745117	5055.712402	-0.000038	0.000182	-18.971739	9784.022461	34505.734375	0.869713	0.000455	2.380542	00:17
79	4728.824219	6232.026855	5679.649414	-0.000018	0.000178	-19.043575	10817.854492	34643.667969	0.920410	0.000449	2.407791	00:16
80	4629.486816	5603.230469	5093.956543	-0.000022	0.000178	-19.134811	9423.280273	34818.835938	0.848731	0.000452	2.481352	00:17
81	4507.021973	5178.119629	4726.227539	-0.000010	0.000157	-19.408560	9442.428711	35344.433594	0.753322	0.000393	2.501068	00:17
82	4538.042480	5305.172852	4842.629883	-0.000003	0.000166	-19.256645	9131.588867	35052.761719	0.770892	0.000414	2.464624	00:17
83	5099.193359	6459.922852	5932.423340	-0.000029	0.000185	-18.995827	10358.086914	34551.984375	0.878512	0.000456	2.431649	00:17
84	4502.932129	4880.916504	4462.495117	-0.000017	0.000155	-19.401627	9218.099609	35331.117188	0.697510	0.000377	2.484270	00:17
85	4412.825684	5199.891113	4784.130859	-0.000022	0.000154	-19.489534	9666.049805	35499.902344	0.692833	0.000379	2.532800	00:16
86	4358.427246	5117.786133	4726.232422	-0.000016	0.000151	-19.556141	9336.509766	35627.792969	0.652101	0.000376	2.560744	00:17
87	4291.551270	4904.972168	4544.766113	-0.000021	0.000138	-19.798647	9187.457031	36093.406250	0.600643	0.000343	2.602123	00:16
88	4245.921387	4626.931641	4291.921875	-0.000019	0.000134	-19.874989	8912.277344	36239.976562	0.558702	0.000335	2.622343	00:16
89	4193.911133	4637.214355	4322.989258	-0.000018	0.000128	-19.985268	8829.379883	36451.714844	0.523725	0.000315	2.629971	00:16
90	4135.915527	5008.456055	4710.114746	-0.000014	0.000120	-20.164143	9413.855469	36795.152344	0.497403	0.000301	2.677485	00:17
91	4054.722168	4418.332520	4167.915039	-0.000017	0.000110	-20.391928	8632.166992	37232.500000	0.417334	0.000273	2.716408	00:17
92	4043.268066	4469.234375	4228.784668	-0.000019	0.000105	-20.496037	8970.399414	37432.394531	0.400793	0.000263	2.740440	00:17
93	3985.776367	4630.074707	4389.567871	-0.000008	0.000103	-20.573969	8729.308594	37582.015625	0.401001	0.000255	2.763188	00:16
94	3928.191162	4238.961914	4040.568115	-0.000013	0.000093	-20.830601	8381.073242	38074.753906	0.330821	0.000230	2.805036	00:17
95	3873.926758	4237.879395	4051.518066	-0.000008	0.000089	-20.973991	8400.993164	38350.070312	0.310790	0.000221	2.853859	00:17
96	3860.987305	4153.635254	3989.159424	-0.000010	0.000082	-21.172232	8493.838867	38730.695312	0.274366	0.000202	2.883892	00:17
97	3856.574463	4109.847656	3941.519043	-0.000010	0.000085	-21.123207	8360.112305	38636.558594	0.280742	0.000211	2.903576	00:17
98	3796.131104	4095.126465	3939.544189	-0.000009	0.000079	-21.274473	8405.658203	38926.996094	0.259483	0.000194	2.915087	00:17
99	3798.427979	4066.241943	3911.510254	-0.000010	0.000078	-21.288141	8324.768555	38953.230469	0.258108	0.000194	2.913828	00:17

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	61706.023438	59401.890625	59200.527344	-0.183162	0.670351	-1.360018	1074.730103	0.182537	64435.050781	1.328313	0.744954	1.561802	00:22
1	58114.496094	54704.398438	54106.003906	-0.067606	0.709372	-1.241548	981.183960	0.604076	61621.859375	1.449838	0.450837	1.893286	00:22
2	55027.734375	52449.945312	52048.351562	-0.016363	0.778369	-0.767656	417.512909	0.947244	60466.074219	0.887459	0.376451	1.181295	00:22
3	53234.660156	51567.859375	51215.320312	-0.014475	0.771869	-0.710858	346.533447	1.000000	59968.601562	0.823350	0.310374	1.014969	00:22
4	52392.648438	51421.828125	51092.546875	-0.048797	0.782629	-0.654477	323.230713	1.000000	59896.394531	0.809924	0.289300	0.944704	00:22

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	73116.570312	56011.906250	41707.714844	-0.162279	0.150928	-43.849857	14298.664062	1.000000	53618.125000	1.220086	0.524610	36.704914	00:22
1	55711.816406	45964.726562	33920.769531	-0.043475	0.279428	-35.963737	12039.008789	1.000000	47239.265625	1.312494	0.757988	37.766373	00:22
2	44836.546875	37305.648438	27406.685547	-0.023731	0.321513	-29.381834	9895.095703	1.000000	41459.550781	1.460417	0.538946	35.474781	00:22
3	36200.839844	30034.455078	21493.714844	-0.048967	0.376191	-24.223383	8544.314453	1.000000	35556.191406	1.591922	0.687644	33.419842	00:22
4	29243.531250	26322.974609	17289.226562	-0.076567	0.462568	-20.003771	9098.589844	1.000000	30504.798828	1.651102	1.249426	31.537554	00:23
5	23948.765625	20693.746094	14159.992188	0.015880	0.476163	-17.972431	6532.034668	1.000000	27060.267578	1.679238	0.613610	30.386091	00:22
6	18914.341797	16118.884766	10100.545898	0.071780	0.485414	-15.699727	6016.059082	1.000000	21769.064453	1.614907	0.813107	28.720329	00:22
7	16302.161133	16133.471680	9065.597656	-0.032543	0.528036	-14.508196	7064.763672	1.000000	20103.945312	1.672818	0.956630	27.728195	00:23
8	14035.663086	13486.658203	8366.876953	-0.014977	0.549360	-13.761081	5118.403809	1.000000	18096.849609	1.596557	0.572698	27.181425	00:23
9	12555.615234	12438.543945	7636.028809	-0.001427	0.551520	-12.375941	4801.065430	1.000000	16884.033203	1.493703	0.722778	25.457499	00:22
10	11842.099609	11512.832031	7183.715820	0.014809	0.537542	-11.792240	4328.155762	1.000000	15679.622070	1.466000	0.521400	24.649998	00:22
11	10849.121094	10924.213867	6975.569824	-0.034562	0.546702	-10.934856	3947.825195	1.000000	15087.824219	1.373056	0.457514	23.260689	00:22
12	10149.152344	10338.746094	6664.760742	0.058513	0.561050	-10.111523	3673.358154	1.000000	14550.804688	1.332573	0.493117	22.023117	00:22
13	9612.157227	9841.376953	6494.279785	-0.028188	0.523342	-9.359609	3346.479980	1.000000	14008.334961	1.275519	0.430882	20.670998	00:22
14	9164.250977	9458.142578	6349.508789	0.017441	0.548174	-8.786793	3108.029785	1.000000	13718.131836	1.209039	0.398604	19.754648	00:22
15	8761.227539	9177.581055	6264.538574	0.044239	0.542927	-8.290083	2912.659668	1.000000	13411.566406	1.172588	0.369216	18.736010	00:22
16	8444.499023	8896.747070	6152.080078	-0.001141	0.557176	-7.828480	2744.129395	1.000000	13307.772461	1.127717	0.398963	17.916582	00:22
17	8178.131836	8776.557617	6145.216309	0.005876	0.548124	-7.555161	2630.786133	1.000000	13287.386719	1.102357	0.376227	17.280994	00:22
18	8179.623535	8713.017578	6129.683105	0.007566	0.553797	-7.419802	2582.790527	1.000000	13186.386719	1.086193	0.387691	17.034954	00:22
19	7979.851562	8669.065430	6088.716309	0.003607	0.549836	-7.418787	2579.850342	1.000000	13166.111328	1.084882	0.384289	17.028463	00:23

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	57399.710938	54946.996094	54929.402344	-0.165153	0.324608	-3.215317	4092.511963	0.003042	62103.660156	3.152873	0.228893	2.357134	00:17
1	51879.070312	47758.117188	47731.417969	-0.061124	0.344856	-2.523539	1779.545044	0.012332	57621.871094	1.957411	0.194889	1.265197	00:17
2	45386.617188	39933.128906	39885.570312	-0.019971	0.228078	-3.189618	1545.098267	0.027755	52250.078125	1.597886	0.104507	0.965323	00:17
3	38245.839844	32358.175781	32285.750000	0.007316	0.198953	-3.397292	1411.495117	0.049117	46108.632812	1.398528	0.077856	0.832803	00:17
4	31773.291016	26649.607422	26547.136719	0.007398	0.220232	-3.210758	1306.681519	0.076149	41807.441406	1.345289	0.088232	0.900020	00:17
5	26294.736328	22049.501953	21913.886719	-0.015827	0.214542	-3.225880	1222.396973	0.108512	37915.671875	1.239124	0.078933	0.783017	00:17
6	21699.671875	18236.214844	18065.015625	0.006644	0.228386	-3.092851	1153.687622	0.145797	34345.519531	1.203612	0.081635	0.765566	00:17
7	17847.849609	15316.773438	15123.155273	0.004208	0.257273	-2.837610	1015.710571	0.187537	31254.556641	1.121719	0.086920	0.723378	00:17
8	14701.538086	12635.338867	12391.008789	-0.013651	0.245611	-2.945722	1035.010498	0.233206	27929.199219	1.102923	0.089294	0.755788	00:18
9	12172.475586	10641.345703	10352.530273	-0.010108	0.255961	-2.870933	1013.395264	0.282230	25045.853516	1.103233	0.098628	0.761307	00:17
10	10187.052734	8859.513672	8534.091797	0.003142	0.269404	-2.769896	967.050171	0.333993	22502.009766	1.079642	0.102157	0.773268	00:17
11	8583.590820	7599.126465	7226.156738	-0.012727	0.276877	-2.742772	956.271484	0.387843	20236.230469	1.073148	0.117234	0.836577	00:17
12	7371.664062	6722.282715	6331.464355	0.001631	0.308230	-2.556443	877.669373	0.443104	18417.957031	1.033615	0.141692	0.907643	00:17
13	6437.998047	5929.011230	5486.416016	0.008249	0.308316	-2.567688	883.479858	0.499080	16502.365234	1.036853	0.142406	0.942082	00:17
14	5715.165527	5563.650879	5117.163574	0.010956	0.338480	-2.387332	801.751221	0.555068	15373.246094	0.988386	0.155976	0.967029	00:18
15	5206.327637	5071.987305	4576.730957	0.018943	0.339977	-2.430615	809.455139	0.610364	13930.858398	0.974587	0.175255	1.070248	00:17
16	4800.470703	4762.033203	4260.530273	0.012815	0.376248	-2.208244	753.381470	0.664271	12816.827148	0.983993	0.184222	1.036388	00:17
17	4534.207031	4585.436523	4030.392334	0.018234	0.373008	-2.265072	774.004639	0.716113	12027.202148	0.986986	0.194114	1.122472	00:17
18	4310.963379	4426.592773	3860.051514	-0.001189	0.383565	-2.204942	739.122742	0.765237	11217.666016	0.956884	0.199072	1.112341	00:17
19	4167.121582	4322.349121	3745.764160	-0.005286	0.411266	-2.095040	709.957886	0.811025	10483.940430	0.950702	0.220418	1.175543	00:17
20	4028.976807	4336.175781	3757.265625	-0.005386	0.437083	-1.981922	677.855469	0.852902	10113.937500	0.943400	0.230930	1.211898	00:17
21	3938.072021	4206.319824	3597.565674	0.018626	0.424841	-2.048255	682.870056	0.890342	9673.818359	0.923458	0.224130	1.237776	00:17
22	3847.404297	4052.002197	3427.098633	-0.000264	0.433609	-2.031101	676.522278	0.922872	9095.830078	0.913549	0.242447	1.261750	00:17
23	3819.536865	4073.962891	3464.709717	0.007187	0.458046	-1.899375	640.617920	0.950085	8923.431641	0.912174	0.242105	1.235491	00:17
24	3762.541260	4000.833008	3355.670166	-0.002811	0.449360	-1.998197	663.433777	0.971637	8458.418945	0.899084	0.250295	1.366378	00:17
25	3736.541260	3943.031006	3347.818604	0.016623	0.487508	-1.806950	602.368286	0.987258	8406.371094	0.875596	0.263050	1.312569	00:17
26	3710.677490	3985.505615	3370.270264	0.007015	0.489873	-1.808489	616.703064	0.996752	8179.481445	0.897920	0.265616	1.337407	00:17
27	3673.440430	3888.875244	3267.376953	-0.002536	0.487086	-1.842913	620.985962	0.999999	8015.021484	0.886212	0.270634	1.376869	00:17
28	3624.921875	3918.490234	3317.113525	-0.011839	0.489002	-1.832910	600.884705	1.000000	7878.039551	0.854293	0.272581	1.370839	00:17
29	3589.374023	3953.299805	3345.688232	0.000822	0.482969	-1.874127	607.109802	1.000000	7959.639648	0.844625	0.273613	1.401480	00:17
30	3563.964844	3815.126465	3219.112061	-0.006048	0.506546	-1.795353	595.567566	1.000000	7648.826172	0.851202	0.288472	1.434759	00:17
31	3526.933350	3777.926270	3180.538574	0.006722	0.498263	-1.818806	596.889709	1.000000	7526.305176	0.848190	0.277884	1.432963	00:18
32	3477.983398	3695.922119	3102.553467	0.005903	0.503670	-1.826391	592.926636	1.000000	7327.599609	0.829604	0.288393	1.481444	00:17
33	3457.067383	3710.122070	3097.999512	0.002594	0.508103	-1.838122	611.681458	1.000000	7163.117676	0.850950	0.299303	1.530949	00:17
34	3423.165527	3636.294922	3047.190674	0.013099	0.509754	-1.816189	588.713440	1.000000	7153.967773	0.821045	0.296447	1.501195	00:17
35	3391.124512	3609.171631	3018.344482	0.003931	0.506914	-1.842947	590.420044	1.000000	7162.170410	0.809100	0.298346	1.531513	00:17
36	3364.496094	3594.141602	3010.026855	0.009317	0.509401	-1.835356	583.693604	1.000000	7114.422852	0.798868	0.299264	1.536168	00:17
37	3330.910889	3557.647217	2969.208496	0.008598	0.509678	-1.848714	588.047607	1.000000	7038.095215	0.797740	0.302051	1.564748	00:17
38	3315.827637	3548.820312	2964.726562	0.006061	0.511136	-1.840202	583.669495	1.000000	7037.770996	0.793503	0.302293	1.559664	00:17
39	3296.066650	3541.617920	2953.627686	0.006044	0.511732	-1.840605	587.596497	1.000000	6985.075684	0.800140	0.303383	1.564387	00:17

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	3575.758301	3683.811279	3101.746826	0.023946	0.515511	-1.756194	581.610535	1.000000	7237.921387	0.842872	0.288098	1.441162	00:17
1	3442.707520	3683.277588	3096.911377	0.021387	0.522397	-1.781124	585.912415	1.000000	7176.047852	0.825728	0.305283	1.536284	00:17
2	3433.988037	3641.492676	3054.416504	-0.011133	0.513312	-1.801112	586.656189	1.000000	7096.828125	0.825289	0.293772	1.515173	00:17
3	3403.571533	3626.560791	3021.381592	0.014340	0.513209	-1.832552	604.736023	1.000000	6905.085449	0.837381	0.302017	1.569364	00:17
4	3390.826904	3625.273193	3048.279785	0.008304	0.522267	-1.787784	576.540222	1.000000	7141.823242	0.803791	0.305938	1.548868	00:17
5	3383.138672	3685.533936	3106.587646	0.017680	0.520357	-1.776290	578.446106	1.000000	7053.821289	0.818500	0.298193	1.524497	00:17
6	3362.476562	3586.350342	3008.823242	0.008792	0.524464	-1.767128	577.107422	1.000000	6952.917480	0.817344	0.303054	1.529652	00:18
7	3348.328613	3592.793701	3004.799072	0.016007	0.519604	-1.800429	587.577087	1.000000	6915.431152	0.820220	0.302051	1.562759	00:17
8	3340.515625	3638.916504	3065.810303	0.007080	0.526563	-1.763072	572.607056	1.000000	6980.000488	0.809834	0.303101	1.543575	00:17
9	3332.447021	3589.719971	2990.395996	0.005587	0.521241	-1.815432	598.889587	1.000000	6889.233887	0.829281	0.308591	1.598692	00:17
10	3324.812012	3572.082275	2991.709717	0.012196	0.529757	-1.778111	579.932251	1.000000	6980.909668	0.809473	0.310917	1.592968	00:17
11	3307.776855	3507.537354	2929.348389	0.003684	0.521830	-1.803354	577.775818	1.000000	6827.665527	0.796723	0.306158	1.582367	00:17
12	3295.577148	3541.513916	2948.527588	0.000647	0.519469	-1.829403	592.570435	1.000000	6840.534180	0.810010	0.308267	1.611026	00:18
13	3306.983398	3521.387695	2918.823730	-0.000043	0.517627	-1.840063	602.151245	1.000000	6716.964355	0.823327	0.307458	1.620270	00:17
14	3294.853027	3530.514160	2947.904541	0.013016	0.523378	-1.793905	582.243469	1.000000	6786.755371	0.811560	0.302712	1.588206	00:17
15	3277.613770	3500.209717	2931.879883	0.003530	0.532002	-1.752866	567.909302	1.000000	6844.532715	0.802326	0.306242	1.569474	00:18
16	3264.204590	3483.020752	2899.177979	0.021454	0.533818	-1.775981	583.373291	1.000000	6674.819824	0.812234	0.317004	1.612443	00:17
17	3248.107422	3581.998535	2971.183350	0.009494	0.511408	-1.879866	610.267090	1.000000	6944.750488	0.817831	0.308653	1.643169	00:17
18	3238.486084	3479.343506	2898.780518	0.001054	0.530051	-1.782171	580.165833	1.000000	6716.494629	0.807055	0.311785	1.604242	00:17
19	3233.876953	3545.872559	2951.255371	0.013293	0.522102	-1.835801	594.115417	1.000000	6775.758789	0.805981	0.311561	1.646345	00:17
20	3233.354980	3508.971436	2906.804443	0.004566	0.524336	-1.820737	601.672974	1.000000	6671.051270	0.827876	0.312659	1.630128	00:17
21	3219.700928	3454.587891	2873.533203	0.015754	0.523143	-1.809343	580.625610	1.000000	6680.046875	0.797551	0.305941	1.613646	00:17
22	3205.050049	3443.900879	2877.469482	0.023677	0.531942	-1.772549	565.960754	1.000000	6833.391113	0.783538	0.311882	1.603098	00:17
23	3196.507812	3444.223877	2859.566162	0.023156	0.529012	-1.802308	584.178284	1.000000	6700.262207	0.801729	0.314129	1.634863	00:17
24	3197.005859	3463.012695	2878.960205	0.003442	0.525198	-1.804367	583.552307	1.000000	6746.915527	0.803448	0.310549	1.603753	00:17
25	3169.403564	3442.520996	2861.700439	0.021838	0.526365	-1.813056	580.388489	1.000000	6809.010742	0.790542	0.311194	1.641972	00:17
26	3118.087402	3361.286133	2744.261475	0.010315	0.508212	-1.883335	616.670410	1.000000	6522.942383	0.831071	0.305663	1.626955	00:18
27	3054.080566	3271.475586	2674.449951	0.008560	0.516198	-1.839113	596.773926	1.000000	6420.662598	0.814873	0.306409	1.609421	00:17
28	3013.062500	3228.558594	2644.365479	0.021629	0.521027	-1.808817	583.952820	1.000000	6468.817871	0.806578	0.303469	1.596425	00:17
29	2980.213623	3298.161865	2682.699951	0.020015	0.511270	-1.893728	615.168823	1.000000	6404.728516	0.817171	0.312471	1.668474	00:17
30	2964.664307	3181.812256	2582.286133	0.004752	0.510557	-1.853738	599.349304	1.000000	6264.407715	0.815534	0.303304	1.592281	00:17
31	2933.877686	3179.411621	2578.830811	0.014322	0.514784	-1.863350	600.360657	1.000000	6315.093750	0.807434	0.306790	1.651939	00:17
32	2911.787109	3220.174561	2639.843262	0.011473	0.528371	-1.776882	580.096436	1.000000	6503.312988	0.811576	0.310953	1.579263	00:17
33	2907.294434	3232.167480	2644.069580	0.001782	0.522584	-1.799896	587.883606	1.000000	6348.228027	0.817487	0.307363	1.582595	00:17
34	2898.118164	3139.234863	2525.521729	0.004105	0.509662	-1.897202	613.484497	1.000000	6183.475586	0.813534	0.311140	1.660771	00:17
35	2886.053223	3146.864746	2556.158936	0.010741	0.521889	-1.818417	590.534058	1.000000	6199.553711	0.810417	0.310111	1.610221	00:17
36	2875.191650	3146.613770	2556.170898	0.005913	0.512729	-1.855228	590.187378	1.000000	6325.440918	0.794197	0.306706	1.609365	00:17
37	2880.403320	3194.668945	2594.254395	0.013286	0.513750	-1.854395	600.159302	1.000000	6224.459961	0.813131	0.307169	1.618436	00:17
38	2875.978760	3183.420654	2564.505615	0.008231	0.506380	-1.906463	618.651001	1.000000	6281.554199	0.821434	0.306852	1.663294	00:17
39	2865.354248	3111.068115	2509.665039	0.012168	0.511287	-1.863821	601.156677	1.000000	6107.499512	0.810342	0.308244	1.612520	00:18
40	2856.117920	3106.837646	2523.184814	0.021064	0.515406	-1.823568	583.439758	1.000000	6323.073242	0.800897	0.301366	1.576838	00:17
41	2841.058350	3224.090088	2620.256592	0.010194	0.511180	-1.869358	603.523315	1.000000	6378.763672	0.811484	0.308618	1.619545	00:17
42	2838.657471	3085.435303	2484.950195	-0.006554	0.512726	-1.864718	600.271240	1.000000	6198.692383	0.806789	0.309381	1.624973	00:17
43	2820.172852	3110.579346	2515.451172	0.017197	0.512071	-1.857899	594.875671	1.000000	6196.029785	0.801827	0.306721	1.609982	00:17
44	2818.890137	3085.406250	2484.689453	0.020887	0.511511	-1.867649	600.557251	1.000000	6122.248047	0.807294	0.306401	1.629988	00:17
45	2821.206543	3125.356201	2516.980713	0.020813	0.510338	-1.877984	608.201660	1.000000	6145.028809	0.815417	0.308699	1.634905	00:17
46	2813.604492	3042.385010	2415.596924	0.016401	0.499427	-1.950168	626.594299	1.000000	5994.569336	0.812704	0.310215	1.676755	00:17
47	2819.581055	3047.210205	2437.243896	0.023057	0.503425	-1.898006	609.807983	1.000000	6060.318359	0.812804	0.302536	1.624642	00:17
48	2803.374268	3052.656738	2444.510498	0.012048	0.510819	-1.874572	607.917725	1.000000	6054.230469	0.817119	0.308117	1.633074	00:17
49	2783.613037	3110.872803	2513.780518	0.007220	0.513580	-1.858175	596.868469	1.000000	6183.766602	0.803926	0.309067	1.624121	00:17
50	2798.391113	3121.154541	2524.978516	0.015719	0.510264	-1.853081	595.888489	1.000000	6336.755371	0.809349	0.303360	1.591298	00:17
51	2787.496338	3005.807373	2404.763672	0.008553	0.509750	-1.859515	600.842834	1.000000	6026.506836	0.815663	0.302265	1.599457	00:17
52	2780.337158	3044.790283	2410.612305	0.023915	0.502508	-1.942298	634.040405	1.000000	5983.610840	0.829220	0.311013	1.694189	00:17
53	2765.546387	3093.744141	2474.235596	0.017311	0.507405	-1.912765	619.238647	1.000000	6051.167480	0.815528	0.312877	1.672780	00:17
54	2771.275146	3040.848633	2445.072998	0.023059	0.511480	-1.852263	595.575256	1.000000	6078.609375	0.808732	0.301422	1.606023	00:17
55	2768.206055	2996.547119	2375.247559	0.007863	0.504959	-1.912884	621.120544	1.000000	5932.541016	0.823699	0.304943	1.670720	00:17
56	2760.632080	3124.823730	2472.626465	0.015785	0.492152	-1.982288	651.971252	1.000000	6025.566895	0.847889	0.303564	1.693272	00:17
57	2762.220215	3061.642578	2449.533447	0.010691	0.509242	-1.889968	611.859863	1.000000	6187.102539	0.816755	0.306674	1.656458	00:17
58	2755.421143	3009.754639	2369.982422	0.000310	0.488715	-1.977646	639.569580	1.000000	5942.870117	0.831614	0.298134	1.657800	00:17
59	2756.570557	2995.561523	2364.336182	0.011086	0.499360	-1.943617	631.053894	1.000000	5944.516602	0.827286	0.304798	1.681580	00:17
60	2757.889160	3027.342773	2398.660889	-0.001433	0.504404	-1.920631	628.486389	1.000000	5940.197266	0.831211	0.310486	1.666582	00:17
61	2746.032471	3007.924316	2376.430664	-0.003403	0.494749	-1.954488	631.278259	1.000000	5961.602539	0.824743	0.302653	1.661404	00:17
62	2738.879150	2970.920898	2363.716553	0.008683	0.504787	-1.883943	607.031006	1.000000	5878.265137	0.816768	0.298905	1.619519	00:18
63	2728.277832	3046.752441	2415.066162	0.006571	0.497172	-1.947274	631.501892	1.000000	5929.144043	0.828103	0.302555	1.668574	00:17
64	2725.899414	2996.217041	2381.287842	0.003617	0.506560	-1.900459	614.756226	1.000000	6011.295410	0.817266	0.307379	1.655017	00:18
65	2716.389160	2961.848145	2342.653320	0.008999	0.501528	-1.912820	618.992310	1.000000	5880.875000	0.823050	0.301267	1.644071	00:18
66	2720.176270	3073.876709	2469.549805	0.010508	0.505660	-1.887435	604.061157	1.000000	6151.004883	0.806969	0.302819	1.625785	00:18
67	2718.482422	2955.271973	2330.459473	-0.004790	0.499610	-1.938555	624.648010	1.000000	5860.532715	0.817574	0.306325	1.666888	00:17
68	2710.359131	2992.895020	2370.543945	0.017971	0.494741	-1.938273	622.101685	1.000000	5885.187988	0.818498	0.298322	1.635081	00:17
69	2703.023682	2947.381104	2332.375244	0.014255	0.503138	-1.906792	614.831787	1.000000	5924.180664	0.816886	0.303687	1.639957	00:17
70	2695.977295	2971.631104	2383.678955	0.020512	0.516536	-1.832951	587.744568	1.000000	6020.938477	0.801660	0.303633	1.612505	00:17
71	2697.532715	2968.887207	2368.074463	0.013142	0.510536	-1.841855	600.604370	1.000000	5890.542480	0.826726	0.298515	1.582411	00:18
72	2702.002686	2975.176514	2360.694580	-0.004192	0.498958	-1.913127	614.294434	1.000000	5982.066895	0.816571	0.298457	1.623456	00:17
73	2703.011475	2963.558594	2333.156738	0.021282	0.492156	-1.947715	630.223572	1.000000	5900.980957	0.829796	0.298024	1.636172	00:18
74	2692.910889	2964.240723	2359.891113	0.009568	0.504947	-1.890549	604.149231	1.000000	5933.749023	0.806391	0.300761	1.631749	00:17
75	2700.743896	2986.143799	2378.706787	0.019656	0.511528	-1.862103	607.162903	1.000000	5877.425781	0.823910	0.304295	1.629194	00:17
76	2686.365967	3032.951172	2409.312988	0.020641	0.506106	-1.900237	623.347168	1.000000	5962.175781	0.833975	0.306373	1.650134	00:17
77	2685.070068	2919.395020	2302.183105	0.018109	0.497977	-1.922022	617.053040	1.000000	5864.688965	0.816511	0.299294	1.633136	00:17
78	2681.744141	2949.721924	2348.547607	0.022095	0.510001	-1.866929	600.984314	1.000000	5912.309570	0.810204	0.304238	1.622940	00:18
79	2664.718750	2905.928467	2287.977295	0.008764	0.495456	-1.927589	617.760986	1.000000	5791.276367	0.816528	0.298251	1.619094	00:17
80	2665.626953	2903.541504	2273.885498	0.013791	0.492093	-1.961995	629.522766	1.000000	5743.782715	0.819196	0.300149	1.658055	00:17
81	2652.062256	2978.843994	2368.964600	0.017189	0.506560	-1.882019	609.651611	1.000000	5891.227051	0.820025	0.303469	1.622615	00:18
82	2650.969482	2891.138916	2264.591064	0.005523	0.494898	-1.949331	626.357666	1.000000	5738.828125	0.818623	0.301926	1.657824	00:17
83	2651.415527	2914.195068	2299.760254	-0.000218	0.499686	-1.913679	614.250916	1.000000	5794.217773	0.815838	0.298123	1.638677	00:17
84	2640.919434	2902.609375	2278.694336	0.020235	0.495504	-1.946337	623.726562	1.000000	5811.029785	0.815382	0.300463	1.660283	00:18
85	2634.171143	2872.147461	2259.986816	0.023257	0.505128	-1.904534	612.018860	1.000000	5757.980957	0.810946	0.304793	1.655894	00:17
86	2616.689453	2901.976074	2285.906982	0.007093	0.504452	-1.916655	615.888489	1.000000	5780.421387	0.810516	0.307877	1.668717	00:17
87	2611.875000	2891.590576	2291.862549	0.016227	0.510713	-1.876446	599.603577	1.000000	5913.339844	0.800907	0.304925	1.652267	00:17
88	2603.816162	2843.069580	2231.054443	0.011312	0.502066	-1.911644	611.847351	1.000000	5739.468750	0.809794	0.300989	1.650247	00:17

epoch	train_loss	valid_loss	l2_b_mean	mmd	mu	std	logvar	l1_b_mean	mu_sd	l1_latent_reg	weighted_kld	logvar_sd	time
0	46222.066406	38895.871094	38823.734375	68.388077	0.027538	1.347189	0.432295	50547.339844	2.096250	6054.318848	9827.441406	0.797124	00:17
1	30034.804688	24725.818359	24648.708984	74.589851	0.094604	1.693746	0.926645	39019.839844	2.388953	6881.258301	13597.906250	0.675845	00:17
2	19339.064453	15980.196289	15887.656250	90.363937	0.099214	2.199340	1.456154	30101.716797	2.903893	8273.364258	22035.894531	0.663303	00:17

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	64307.953125	58279.812500	58243.433594	-0.050396	0.446575	-1.720978	633.586060	0.047932	64056.988281	0.950300	0.219880	0.624841	00:24
1	52256.531250	47517.250000	47401.242188	-0.095860	0.402663	-2.017131	589.741516	0.185387	58108.828125	0.767923	0.194185	0.970138	00:23
2	34960.160156	31226.734375	30933.964844	-0.076051	0.269091	-2.800119	751.623657	0.385156	45276.707031	0.674353	0.099892	0.940919	00:24
3	24776.472656	20445.687500	20104.564453	-0.063708	0.384149	-2.067982	556.550415	0.607671	36356.878906	0.703798	0.140935	0.863217	00:23
4	17783.804688	16285.233398	15865.710938	-0.067235	0.429491	-1.891830	517.371826	0.808860	31224.277344	0.709278	0.170626	0.993864	00:24
5	13278.303711	11126.129883	10704.387695	-0.060666	0.502716	-1.599754	443.023224	0.948876	24944.644531	0.696404	0.200470	1.058257	00:24
6	10607.653320	9407.301758	8986.120117	-0.058500	0.538358	-1.487504	420.064575	0.999986	22220.373047	0.692390	0.222120	1.125344	00:24
7	9076.883789	8579.513672	8182.976074	-0.048066	0.568540	-1.401409	394.949036	1.000000	20782.205078	0.669947	0.237349	1.191771	00:24
8	8298.398438	8123.069824	7729.753418	-0.048600	0.572593	-1.406226	391.598938	1.000000	20188.673828	0.652731	0.242710	1.237344	00:24
9	7958.245605	8007.626953	7610.983887	-0.048821	0.575821	-1.404351	394.969757	1.000000	19997.000000	0.657966	0.246665	1.262074	00:24

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	7446.696289	7002.534668	6594.965820	-0.056031	0.609424	-1.249221	406.261200	1.000000	16749.544922	0.762242	0.250232	1.164957	00:24
1	6559.509766	6201.054688	5791.714844	-0.041461	0.618311	-1.230625	408.183350	1.000000	14291.185547	0.769989	0.256279	1.197002	00:24
2	5965.905762	5736.044434	5315.900879	-0.065841	0.617567	-1.259662	419.067261	1.000000	13115.260742	0.771636	0.256352	1.277502	00:24
3	5546.066406	5484.937012	5098.352539	-0.031883	0.655282	-1.149140	385.752136	1.000000	12599.239258	0.741821	0.269175	1.304524	00:23
4	5216.171387	5357.946289	4967.701172	-0.048715	0.659577	-1.167782	389.449951	1.000000	11821.793945	0.728742	0.278477	1.384185	00:24
5	4968.410156	4974.411133	4575.067383	-0.037054	0.655091	-1.190241	398.651703	1.000000	11053.794922	0.738295	0.278551	1.407455	00:24
6	4758.382812	4930.711914	4530.911133	-0.042303	0.655234	-1.206222	399.300415	1.000000	10765.223633	0.727668	0.281477	1.447791	00:24
7	4598.325684	4773.492188	4384.264648	-0.028840	0.662661	-1.158184	388.670288	1.000000	10158.075195	0.732503	0.278200	1.385599	00:24
8	4481.006348	4676.386719	4256.562988	-0.047947	0.652907	-1.256124	419.240143	1.000000	9864.458008	0.734679	0.288794	1.560645	00:24
9	4396.450195	4542.811035	4142.692871	-0.031679	0.664870	-1.194383	399.589661	1.000000	9653.433594	0.725160	0.288026	1.493910	00:23
10	4277.703125	4514.132324	4093.090332	-0.038448	0.659772	-1.263761	420.567749	1.000000	9540.075195	0.722800	0.298543	1.615734	00:24
11	4211.627441	4451.823730	4059.461670	-0.017897	0.679049	-1.174503	391.879242	1.000000	9462.656250	0.704454	0.300546	1.540585	00:23
12	4159.201660	4370.717285	3985.004150	-0.029835	0.686301	-1.132567	385.226257	1.000000	9333.199219	0.711258	0.301613	1.484894	00:24
13	4089.183594	4338.795898	3938.385986	-0.027388	0.671400	-1.209984	399.877716	1.000000	9139.356445	0.703476	0.301768	1.563450	00:23
14	4024.176514	4243.605957	3844.971924	-0.038013	0.681120	-1.182844	398.243713	1.000000	8863.547852	0.705809	0.307898	1.564476	00:24
15	3992.836426	4182.538574	3768.346436	-0.015892	0.671695	-1.240414	413.820831	1.000000	8815.115234	0.708553	0.310647	1.627596	00:24
16	3938.035645	4254.535156	3836.140381	-0.021022	0.672339	-1.260805	417.951782	1.000000	8692.565430	0.700489	0.314746	1.681814	00:24
17	3915.346924	4136.510254	3734.750488	-0.037996	0.684208	-1.182841	401.324951	1.000000	8577.833008	0.708467	0.312179	1.580640	00:24
18	3851.800781	4099.753418	3683.509521	-0.035467	0.673168	-1.264503	415.946136	1.000000	8637.051758	0.690829	0.317606	1.686912	00:24
19	3829.080811	4061.842773	3656.622803	-0.020944	0.678215	-1.224614	404.792664	1.000000	8405.791992	0.691823	0.315146	1.633282	00:24
20	3814.291260	4050.546143	3625.365479	-0.017918	0.671568	-1.263860	424.740845	1.000000	8314.105469	0.712885	0.319519	1.662932	00:24
21	3790.595947	4220.412598	3789.602051	-0.022943	0.678726	-1.280803	430.250519	1.000000	8668.383789	0.700995	0.329524	1.751208	00:23
22	3769.847412	4021.063232	3616.341309	-0.031142	0.680278	-1.221621	404.374634	1.000000	8211.235352	0.687985	0.318570	1.633209	00:24
23	3756.498779	3959.629883	3536.583008	-0.020466	0.670928	-1.292554	422.621613	1.000000	8118.128418	0.686245	0.323532	1.728216	00:24
24	3706.817871	4126.592773	3708.841553	-0.023985	0.678128	-1.259123	417.260498	1.000000	8252.971680	0.689504	0.326434	1.692541	00:24
25	3687.163330	4039.027832	3624.490723	-0.023506	0.679744	-1.252728	414.078033	1.000000	8196.382812	0.684681	0.328139	1.685571	00:24
26	3664.687012	3962.185303	3553.306396	-0.020275	0.681983	-1.241518	408.559418	1.000000	8060.037109	0.678305	0.328026	1.677929	00:24
27	3655.794189	3935.266602	3520.048096	-0.026080	0.684890	-1.246303	414.766052	1.000000	8066.496094	0.683167	0.332905	1.703669	00:24
28	3633.048096	3900.002441	3477.207520	-0.031444	0.678460	-1.280428	422.386871	1.000000	7979.427734	0.682424	0.332304	1.739592	00:23
29	3605.916504	3902.634277	3490.624512	-0.027529	0.688832	-1.218063	411.608704	1.000000	7930.282715	0.693239	0.331436	1.667827	00:23
30	3597.799072	3864.459473	3440.049072	-0.031359	0.680125	-1.279481	424.118408	1.000000	7889.754883	0.684454	0.334076	1.745801	00:23
31	3577.369385	3835.365479	3421.598145	-0.031690	0.687918	-1.243465	413.432159	1.000000	7764.869629	0.677153	0.336913	1.711155	00:23
32	3567.163574	3808.568115	3392.390869	-0.030360	0.685564	-1.245607	415.832184	1.000000	7823.952148	0.684545	0.334472	1.705133	00:24
33	3556.911133	3839.799316	3426.487305	-0.025034	0.679174	-1.259462	412.899506	1.000000	8020.717285	0.675566	0.331511	1.691069	00:23
34	3547.733643	3787.851562	3366.338867	-0.033549	0.684411	-1.252932	421.170959	1.000000	7632.813965	0.692167	0.335533	1.710420	00:23
35	3498.793457	3750.230957	3341.822998	-0.024690	0.690246	-1.227408	408.039459	1.000000	7705.300781	0.674841	0.336464	1.692907	00:24
36	3497.622314	3779.739746	3356.960205	-0.031387	0.678122	-1.298752	422.325592	1.000000	7688.250488	0.666296	0.338358	1.765067	00:23
37	3518.062500	3770.506348	3337.693848	-0.026667	0.684909	-1.278552	432.439392	1.000000	7656.469727	0.695298	0.342245	1.764647	00:24
38	3493.117676	3758.422607	3342.844238	-0.021858	0.688443	-1.241507	415.205994	1.000000	7605.489746	0.681559	0.339791	1.703853	00:24
39	3465.591064	3732.348145	3307.750488	-0.019985	0.681497	-1.268380	424.199097	1.000000	7533.059082	0.689752	0.338435	1.715226	00:24
40	3459.664551	3786.777588	3376.685059	-0.013549	0.681292	-1.238933	409.652313	1.000000	7627.532227	0.682110	0.330924	1.658110	00:23
41	3466.934326	3731.796387	3311.013428	-0.029070	0.678285	-1.272447	420.435974	1.000000	7626.694824	0.682827	0.335078	1.704524	00:23
42	3442.351074	3738.564697	3302.626465	-0.041683	0.680691	-1.311230	435.543732	1.000000	7666.170898	0.680946	0.344484	1.805883	00:24
43	3429.302246	3697.198730	3261.440674	-0.023874	0.680301	-1.309720	435.364044	1.000000	7565.525879	0.682782	0.345298	1.794736	00:24
44	3421.857422	3716.597656	3288.889404	-0.018520	0.678248	-1.302927	427.315948	1.000000	7654.982910	0.672833	0.342392	1.766432	00:23
45	3432.586670	3763.864990	3343.723389	-0.013860	0.682215	-1.270576	419.691833	1.000000	7661.630859	0.677036	0.339797	1.728749	00:24
46	3409.428711	3733.609619	3305.357666	-0.026532	0.675712	-1.306996	427.861542	1.000000	7548.878906	0.673747	0.341761	1.756524	00:24
47	3396.132812	3671.088623	3244.154297	-0.027941	0.680358	-1.281522	426.605988	1.000000	7443.862793	0.685382	0.340985	1.736874	00:24
48	3388.988281	3633.996582	3204.664795	-0.028510	0.683743	-1.284135	428.964844	1.000000	7269.158203	0.683487	0.344635	1.762939	00:23
49	3374.890625	3655.472168	3228.736816	-0.025874	0.681976	-1.279506	426.387177	1.000000	7347.823730	0.683649	0.342960	1.738811	00:24
50	3374.364014	3661.433594	3226.835205	-0.027927	0.680691	-1.301747	434.195587	1.000000	7426.177734	0.685498	0.345380	1.777534	00:24
51	3377.024414	3646.464844	3221.161377	-0.040788	0.683618	-1.276678	424.889130	1.000000	7338.713379	0.679540	0.343833	1.744919	00:24
52	3367.656982	3639.472168	3208.727295	-0.026861	0.677835	-1.289724	430.378632	1.000000	7426.609863	0.690034	0.342502	1.726464	00:24
53	3353.731689	3598.407715	3163.959717	-0.032011	0.680891	-1.294131	434.070862	1.000000	7290.533203	0.690069	0.345623	1.758445	00:24
54	3350.532715	3618.042969	3183.317627	-0.024679	0.679061	-1.301106	434.440765	1.000000	7441.650879	0.688150	0.345228	1.761177	00:24
55	3328.891113	3591.654785	3151.659912	-0.024667	0.675619	-1.320078	439.684174	1.000000	7220.856445	0.689841	0.345016	1.782546	00:24
56	3320.435059	3628.756104	3184.838135	-0.023147	0.673282	-1.327310	443.469299	1.000000	7381.559082	0.695388	0.344838	1.779874	00:24
57	3319.204590	3589.929688	3161.705322	-0.029322	0.681353	-1.283165	427.894348	1.000000	7210.187988	0.684854	0.343528	1.745058	00:24
58	3333.072510	3605.556396	3159.020752	-0.022901	0.669782	-1.351565	446.191711	1.000000	7292.311523	0.687392	0.345130	1.809493	00:24
59	3301.860840	3583.256592	3147.702881	-0.018526	0.682041	-1.308480	435.160400	1.000000	7301.224609	0.678834	0.351069	1.792501	00:23
60	3289.162598	3571.066650	3126.065918	-0.025570	0.675508	-1.345270	444.585358	1.000000	7098.305664	0.679848	0.351047	1.829405	00:23
61	3285.661377	3577.617432	3150.016113	-0.022347	0.678402	-1.289324	427.211182	1.000000	7195.916504	0.681604	0.343973	1.729821	00:24
62	3292.790527	3545.149658	3106.085693	-0.020378	0.672821	-1.324593	438.766663	1.000000	7157.503418	0.687220	0.345001	1.767740	00:24
63	3273.931641	3576.871826	3127.184814	-0.030022	0.663642	-1.371861	449.397675	1.000000	7171.856934	0.685767	0.344284	1.808998	00:24
64	3286.415527	3542.644287	3110.177246	-0.025038	0.680133	-1.290051	432.100250	1.000000	7128.390137	0.689775	0.345855	1.743150	00:24
65	3295.292969	3554.997070	3110.141602	-0.031996	0.673540	-1.316412	444.468079	1.000000	7076.810547	0.704291	0.345098	1.755248	00:24
66	3287.177979	3522.089844	3080.620850	-0.039147	0.673940	-1.317083	441.162537	1.000000	7100.459961	0.695908	0.345708	1.758326	00:24
67	3241.864502	3488.717773	3024.696045	-0.025456	0.659230	-1.414685	463.702454	1.000000	7130.245605	0.688320	0.351045	1.855794	00:23
68	3192.238281	3457.201172	3000.528809	-0.025450	0.664398	-1.374820	456.514099	1.000000	6979.332031	0.696865	0.349181	1.810351	00:24
69	3159.333496	3475.097168	3000.846436	-0.023379	0.655233	-1.422044	474.188934	1.000000	7015.226074	0.712399	0.347041	1.856408	00:23
70	3117.958740	3506.858643	3045.154053	-0.011474	0.658447	-1.394947	461.412323	1.000000	7187.719727	0.700233	0.347915	1.812438	00:24
71	3089.015869	3462.144531	3013.021973	-0.024864	0.666162	-1.344185	448.933685	1.000000	7037.438965	0.701362	0.347054	1.756467	00:23
72	3063.737793	3324.433838	2869.878662	-0.035118	0.660016	-1.366300	454.371368	1.000000	6876.746582	0.703546	0.345587	1.760979	00:24
73	3027.214111	3331.492920	2870.182617	-0.011670	0.660568	-1.397654	461.116272	1.000000	6893.470215	0.693894	0.351798	1.828764	00:24
74	3017.817139	3321.054199	2865.548340	-0.025422	0.660219	-1.382914	455.403198	1.000000	6842.084473	0.691309	0.352129	1.784368	00:24
75	2992.729736	3256.467041	2794.733398	-0.019993	0.661402	-1.395585	461.590149	1.000000	6778.131348	0.693400	0.355003	1.819794	00:23
76	2983.455566	3289.244873	2831.101807	-0.029130	0.667984	-1.363975	457.947998	1.000000	6734.300781	0.701110	0.355252	1.800379	00:23
77	2976.864746	3239.218018	2781.242432	-0.028557	0.660307	-1.386050	457.947540	1.000000	6727.701172	0.694157	0.352464	1.793653	00:23
78	2956.301025	3304.631104	2842.858887	-0.027479	0.657182	-1.408286	461.591095	1.000000	6794.987793	0.687993	0.355320	1.808959	00:24
79	2945.924316	3334.847900	2866.822021	-0.024711	0.658704	-1.412751	467.761139	1.000000	6867.231934	0.696831	0.356441	1.834160	00:23
80	2924.367188	3198.714355	2731.728760	-0.018036	0.659534	-1.423019	466.900421	1.000000	6670.595703	0.685548	0.359238	1.858049	00:24
81	2906.989014	3215.365967	2753.775635	-0.021889	0.660180	-1.399861	461.448822	1.000000	6624.345215	0.690112	0.357600	1.810584	00:24
82	2896.910645	3215.521729	2746.866455	-0.023133	0.657917	-1.429965	468.538452	1.000000	6574.671387	0.685399	0.359657	1.859299	00:24
83	2868.951416	3202.712891	2736.927490	-0.030787	0.663033	-1.420170	465.648895	1.000000	6593.380371	0.678296	0.363546	1.870697	00:23
84	2858.278809	3162.275635	2696.020264	-0.019797	0.660603	-1.421584	466.094910	1.000000	6575.957520	0.682723	0.360896	1.858423	00:24
85	2846.207275	3103.525635	2629.209473	-0.024866	0.656847	-1.462483	474.175415	1.000000	6484.671875	0.671980	0.365991	1.908332	00:24
86	2817.036377	3078.863525	2616.457520	-0.022164	0.662941	-1.424628	462.297211	1.000000	6458.416992	0.666711	0.365627	1.872652	00:24
87	2798.375732	3076.187744	2600.413330	-0.032358	0.658778	-1.468475	475.640625	1.000000	6369.106445	0.666486	0.369961	1.928813	00:24
88	2771.174316	3070.136230	2606.374756	-0.024365	0.665567	-1.433004	463.692261	1.000000	6426.307617	0.658267	0.370472	1.902251	00:24
89	2762.876221	3072.244873	2603.819824	-0.026865	0.662920	-1.453525	468.355865	1.000000	6418.618652	0.655338	0.372392	1.920821	00:23
90	2738.615723	3011.625732	2538.651123	-0.022828	0.662585	-1.463570	472.897278	1.000000	6366.740723	0.658284	0.373876	1.937470	00:23
91	2722.517822	2995.657715	2518.491211	-0.018094	0.661352	-1.488024	477.079224	1.000000	6295.547852	0.649196	0.376828	1.975732	00:24
92	2708.709961	3014.197021	2543.201416	-0.019909	0.665528	-1.464587	470.887909	1.000000	6312.110840	0.647752	0.377463	1.954773	00:24
93	2697.568115	2966.013672	2492.254150	-0.020296	0.664736	-1.472951	473.693115	1.000000	6222.543945	0.648153	0.378263	1.965246	00:23
94	2692.429199	2959.528076	2484.598633	-0.019818	0.663212	-1.485151	474.830475	1.000000	6250.688965	0.642955	0.378584	1.979259	00:23
95	2679.370361	2948.991699	2474.185791	-0.017727	0.664275	-1.484554	474.726776	1.000000	6253.794922	0.641554	0.379589	1.984106	00:24
96	2667.490234	2942.266357	2468.315674	-0.019934	0.664913	-1.478830	473.874390	1.000000	6198.797852	0.643243	0.379516	1.976109	00:24
97	2663.017822	2939.738770	2463.631104	-0.018928	0.664092	-1.489275	476.029968	1.000000	6207.451660	0.640717	0.380436	1.990803	00:24
98	2661.197754	2935.853516	2459.528320	-0.019907	0.664481	-1.487758	476.260712	1.000000	6187.456543	0.641910	0.380615	1.990031	00:24
99	2662.050293	2935.732422	2458.348145	-0.019693	0.665127	-1.483711	477.305267	1.000000	6187.085449	0.647012	0.380451	1.986876	00:24

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	57119.871094	48978.390625	48411.449219	-0.171113	0.467323	-1.606784	11681.308594	0.047932	57218.921875	4.143692	0.132895	0.589939	00:25
1	42990.976562	33256.042969	32888.765625	-0.111939	0.787082	-0.534524	1944.819458	0.185387	45751.316406	1.667504	0.184117	0.476903	00:25
2	27649.550781	18003.472656	17567.388672	-0.051287	0.614378	-1.090928	1118.868530	0.385156	32650.421875	1.105906	0.211503	0.686238	00:25
3	18592.869141	13246.469727	12702.623047	-0.040173	0.588392	-1.146373	889.270142	0.607671	27025.304688	0.926397	0.170838	0.591538	00:25
4	13824.781250	11302.725586	10674.322266	-0.025407	0.626357	-1.017098	773.015381	0.808860	24171.271484	0.872669	0.170972	0.585883	00:25
5	11246.221680	9732.048828	9055.437500	0.002667	0.623677	-1.060219	710.477600	0.948876	21821.728516	0.787523	0.195109	0.716930	00:25
6	9790.408203	9052.549805	8347.827148	0.037016	0.624793	-1.088983	702.386963	0.999986	20620.949219	0.755349	0.210199	0.831470	00:25
7	8893.800781	8546.672852	7901.442383	0.032866	0.649115	-1.026540	642.988525	1.000000	19937.876953	0.710113	0.222703	0.880089	00:25
8	8421.801758	8350.280273	7711.315918	0.032067	0.647427	-1.042604	636.768677	1.000000	19623.410156	0.692194	0.226795	0.915346	00:25
9	8228.472656	8303.927734	7656.038574	0.032340	0.642907	-1.063508	645.732849	1.000000	19511.349609	0.690796	0.228713	0.935445	00:25

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	7636.996094	7720.530273	7083.126953	0.019008	0.683685	-0.985189	635.105652	1.000000	17760.312500	0.687582	0.255679	1.082731	00:26
1	6885.542480	6894.079590	6297.856445	0.006627	0.707549	-0.936249	594.373535	1.000000	16222.292969	0.647742	0.266870	1.149313	00:25
2	6309.057617	6247.788574	5663.020508	0.013977	0.733179	-0.899275	583.153992	1.000000	14961.074219	0.627324	0.282289	1.256484	00:25
3	5862.837891	5824.015137	5272.883301	0.018640	0.760227	-0.806429	549.852478	1.000000	13915.848633	0.627636	0.282030	1.206542	00:25
4	5524.013672	5580.453613	4991.576660	0.000202	0.744082	-0.900245	587.568359	1.000000	13173.700195	0.615521	0.290766	1.348567	00:25
5	5247.176758	5352.265625	4772.904785	0.012758	0.746673	-0.887565	578.224121	1.000000	12680.643555	0.611426	0.288877	1.332583	00:25
6	5000.711914	5149.272949	4567.143555	0.021369	0.757468	-0.872349	581.102356	1.000000	12148.317383	0.609617	0.297950	1.367787	00:25
7	4822.857422	4957.025879	4346.383789	0.012121	0.743142	-0.944743	609.705933	1.000000	11592.447266	0.603416	0.300536	1.456099	00:25
8	4670.200195	4811.929199	4201.861328	0.023626	0.744356	-0.943792	609.286682	1.000000	11109.132812	0.600485	0.303331	1.457995	00:25
9	4552.351562	4701.762207	4124.187012	0.012166	0.752433	-0.885569	576.840942	1.000000	10700.002930	0.601379	0.294862	1.367969	00:25
10	4457.607422	4782.782715	4139.709961	0.030688	0.750395	-0.973056	642.217896	1.000000	10746.132812	0.605009	0.315268	1.573785	00:25
11	4389.464844	4600.456543	3989.061279	0.006628	0.754086	-0.906970	610.690735	1.000000	10181.538086	0.620296	0.304632	1.429988	00:25
12	4316.668945	4546.292480	3929.669189	0.008763	0.755007	-0.928994	615.957153	1.000000	10081.615234	0.603556	0.313128	1.485501	00:26
13	4249.416016	4501.651855	3910.774902	0.001233	0.753363	-0.902888	590.159973	1.000000	9931.499023	0.598335	0.304420	1.410765	00:25
14	4194.772461	4419.601562	3797.657227	0.011631	0.748259	-0.947026	621.403442	1.000000	9608.806641	0.604558	0.311978	1.482321	00:26
15	4155.236328	4390.953125	3742.381348	0.013369	0.749580	-0.965407	648.057068	1.000000	9297.745117	0.618429	0.318911	1.533816	00:25
16	4107.301758	4357.127441	3726.751709	0.014419	0.751795	-0.950445	629.798523	1.000000	9370.619141	0.605639	0.317125	1.514558	00:25
17	4080.357178	4326.577148	3693.467285	0.019456	0.757275	-0.930076	632.673401	1.000000	9206.707031	0.618530	0.317690	1.501742	00:25
18	4047.259766	4325.830566	3670.197754	0.016036	0.747400	-0.985562	655.035889	1.000000	9117.528320	0.611593	0.321978	1.569723	00:25
19	4017.182617	4268.327637	3646.175049	0.016499	0.751013	-0.935652	621.611816	1.000000	9072.065430	0.609426	0.314141	1.470098	00:25
20	3994.158936	4206.021484	3567.909668	0.020197	0.751552	-0.950140	637.651001	1.000000	8851.089844	0.615019	0.318669	1.509441	00:25
21	3978.268555	4256.660156	3621.489990	0.006194	0.745934	-0.969727	634.768066	1.000000	8992.529297	0.602167	0.318927	1.514271	00:26
22	3955.441650	4265.469238	3629.651123	-0.002801	0.752090	-0.953087	635.321167	1.000000	8820.404297	0.607925	0.321245	1.517332	00:26
23	3929.008545	4206.423828	3560.397217	0.014343	0.746978	-0.970275	645.613831	1.000000	8713.512695	0.614093	0.319624	1.526678	00:25
24	3908.563965	4153.493164	3523.141113	0.006290	0.751558	-0.946957	629.941040	1.000000	8658.349609	0.608749	0.317679	1.503032	00:25
25	3898.939697	4141.995605	3511.120850	0.015062	0.746144	-0.961228	630.455627	1.000000	8498.142578	0.604333	0.316788	1.497967	00:25
26	3885.578369	4103.011230	3454.336182	0.016320	0.742984	-0.985140	648.239014	1.000000	8472.931641	0.609011	0.321561	1.529650	00:25
27	3885.664307	4144.894043	3514.354492	0.007924	0.749983	-0.937831	630.100952	1.000000	8521.073242	0.619199	0.315660	1.465887	00:25
28	3863.140381	4171.204590	3528.229492	0.008750	0.749470	-0.969313	642.520569	1.000000	8679.597656	0.605489	0.324102	1.536831	00:25
29	3844.579834	4065.047852	3432.645752	0.014242	0.751757	-0.957529	631.998291	1.000000	8357.607422	0.599753	0.322359	1.528431	00:25
30	3806.262695	4071.528809	3425.390625	0.006845	0.749838	-0.974802	645.797546	1.000000	8272.665039	0.603490	0.326274	1.551809	00:25
31	3803.645508	4030.028564	3395.478516	0.018196	0.753908	-0.944004	634.221741	1.000000	8226.596680	0.611156	0.321838	1.508687	00:25
32	3798.153320	4027.861572	3400.078369	0.002770	0.757909	-0.921416	627.355713	1.000000	8197.963867	0.616403	0.321067	1.476825	00:25
33	3795.605469	4068.377686	3419.073730	-0.000919	0.747791	-0.983250	648.834473	1.000000	8331.038086	0.603142	0.326154	1.559857	00:26
34	3774.649170	4009.320557	3368.954346	0.006577	0.745278	-0.975523	640.058167	1.000000	8151.725098	0.603200	0.322254	1.523311	00:25
35	3797.206055	4070.147705	3423.396484	0.010856	0.747066	-0.977548	646.342896	1.000000	8184.187500	0.606127	0.324828	1.541001	00:25
36	3764.414795	3997.911377	3347.640381	0.000495	0.740897	-0.989360	649.856018	1.000000	8078.457031	0.609805	0.322489	1.524083	00:25
37	3738.304443	4000.742188	3355.457764	0.006559	0.746498	-0.979872	644.819946	1.000000	8064.453125	0.602224	0.326189	1.537351	00:25
38	3721.401367	4012.916016	3346.493408	0.014446	0.741339	-0.996954	665.997559	1.000000	7987.633789	0.622092	0.324676	1.547024	00:25
39	3733.728516	4002.178223	3343.464111	0.003242	0.744426	-0.993874	658.359314	1.000000	8037.950195	0.610395	0.326803	1.561654	00:25
40	3713.223877	3972.769287	3311.979980	0.027494	0.745946	-0.984094	660.406372	1.000000	7864.602051	0.618499	0.326929	1.545728	00:25
41	3725.438721	4000.221680	3349.538574	0.010210	0.749517	-0.965508	650.194214	1.000000	7884.955078	0.617907	0.324975	1.529040	00:25
42	3714.680908	4006.921143	3337.395020	0.006614	0.741850	-1.008360	669.165833	1.000000	7901.441406	0.614372	0.328160	1.577131	00:25
43	3684.118408	3953.133789	3283.846680	0.002745	0.745630	-1.007263	668.936707	1.000000	7932.915527	0.609230	0.331397	1.602322	00:25
44	3617.731934	3961.301758	3277.286377	0.008253	0.732352	-1.054439	683.772156	1.000000	8140.794434	0.605360	0.331876	1.616396	00:25
45	3527.310547	3771.926025	3082.391357	0.021190	0.732152	-1.045097	689.335327	1.000000	7549.060059	0.621409	0.328936	1.597990	00:25
46	3481.053223	3792.558594	3111.916016	0.014146	0.736934	-1.022333	680.446411	1.000000	7618.551758	0.623152	0.328771	1.574195	00:25
47	3449.490723	3723.544434	3044.811035	0.004227	0.736182	-1.023490	678.525024	1.000000	7445.786133	0.619757	0.330939	1.563343	00:26
48	3453.458984	3735.135986	3037.158936	0.002894	0.727846	-1.065144	697.809265	1.000000	7540.457520	0.619377	0.332484	1.605249	00:25
49	3440.742188	3698.924316	3023.513428	0.015087	0.739747	-1.005747	675.299194	1.000000	7419.930664	0.626200	0.329499	1.548569	00:25
50	3408.526855	3675.649902	3002.518066	0.007204	0.733653	-1.024227	672.936035	1.000000	7433.500000	0.616485	0.328433	1.547451	00:25
51	3414.460693	3745.192383	3054.504639	0.023048	0.737199	-1.023411	690.448914	1.000000	7412.087402	0.632781	0.331396	1.572160	00:25
52	3390.644287	3662.520020	2988.273438	0.007120	0.741236	-1.006877	674.001587	1.000000	7432.633301	0.620694	0.332220	1.557971	00:25
53	3382.684326	3674.317383	3003.882568	-0.000215	0.739698	-1.012987	670.337097	1.000000	7319.799805	0.612742	0.332873	1.558902	00:25
54	3380.493896	3745.933105	3057.020508	0.010614	0.740175	-1.024149	688.698853	1.000000	7467.467773	0.624081	0.336852	1.585891	00:25
55	3365.373047	3633.520752	2938.203613	0.000266	0.735362	-1.048191	695.175659	1.000000	7304.637695	0.618786	0.336548	1.616588	00:25
56	3360.692871	3634.707031	2946.971191	0.006276	0.732114	-1.034212	687.640747	1.000000	7217.326172	0.627182	0.330707	1.557482	00:25
57	3361.901367	3677.013184	2977.446289	0.008722	0.729319	-1.064693	699.587585	1.000000	7371.806152	0.619002	0.334934	1.612427	00:26
58	3341.074951	3619.520508	2932.792236	0.004525	0.736755	-1.030560	686.562866	1.000000	7233.354980	0.621963	0.333719	1.583878	00:25
59	3341.017334	3677.554688	2991.288574	-0.001353	0.736229	-1.028123	686.075989	1.000000	7211.670410	0.624420	0.332879	1.574305	00:25
60	3336.556885	3614.616699	2914.052734	0.012258	0.729867	-1.065539	700.479431	1.000000	7202.227539	0.618801	0.334756	1.621663	00:26
61	3338.885986	3666.255859	2975.747559	0.011388	0.735329	-1.025145	690.370422	1.000000	7230.252930	0.633762	0.331731	1.559722	00:25
62	3316.322510	3614.233643	2910.035645	0.011739	0.730760	-1.060256	703.997192	1.000000	7130.198242	0.625956	0.335665	1.610752	00:25
63	3306.802246	3591.011963	2898.936035	0.019578	0.729113	-1.051367	692.025085	1.000000	7151.365234	0.622282	0.331209	1.580528	00:25
64	3319.408691	3607.750244	2936.093506	0.015589	0.738210	-1.010526	671.561401	1.000000	7256.174805	0.619674	0.329484	1.550395	00:25
65	3329.283447	3666.196533	2997.034180	0.007939	0.738448	-1.005233	669.044373	1.000000	7284.672363	0.620451	0.329718	1.533679	00:25
66	3319.029053	3617.955322	2919.307373	0.009864	0.730675	-1.044238	698.430969	1.000000	7143.467285	0.634253	0.331163	1.574749	00:25
67	3304.009033	3641.074951	2941.493896	0.036581	0.734014	-1.039964	699.558655	1.000000	7150.225098	0.633657	0.332079	1.591718	00:25
68	3283.734131	3604.605469	2921.168701	0.009594	0.734629	-1.032366	683.271362	1.000000	7177.411621	0.619841	0.331710	1.576211	00:25
69	3282.202637	3547.863525	2858.945801	0.018959	0.732195	-1.035658	688.809265	1.000000	7019.934570	0.627719	0.329841	1.567864	00:26
70	3285.655762	3639.036133	2936.884277	0.013103	0.730776	-1.055586	701.959717	1.000000	7140.610840	0.627760	0.334509	1.601171	00:26
71	3288.370117	3591.168457	2893.040771	0.015652	0.726576	-1.061164	697.990662	1.000000	7123.018555	0.624936	0.331148	1.586430	00:25
72	3270.113281	3567.283447	2889.191162	0.009472	0.737333	-1.018427	677.971008	1.000000	7061.854004	0.621693	0.330927	1.562181	00:25
73	3268.113281	3598.905273	2899.352539	0.011853	0.725724	-1.065427	699.484924	1.000000	7144.051270	0.624485	0.331081	1.592555	00:25
74	3265.667236	3563.320068	2867.907227	0.012042	0.729498	-1.047444	695.306763	1.000000	7025.574219	0.628830	0.331620	1.572838	00:25
75	3245.814453	3517.445068	2836.110107	0.003315	0.734034	-1.036802	681.251587	1.000000	7037.501953	0.613656	0.333098	1.580311	00:25
76	3241.935303	3530.926758	2817.048340	0.011109	0.723914	-1.084425	713.788513	1.000000	6924.078125	0.627887	0.333674	1.624286	00:25
77	3245.354980	3546.132568	2842.161621	0.005974	0.725738	-1.074948	703.946472	1.000000	6994.406738	0.621140	0.333325	1.616471	00:25
78	3245.945557	3528.050781	2828.606201	0.012258	0.728167	-1.062054	699.372986	1.000000	6909.879883	0.623076	0.333196	1.599359	00:25
79	3229.143799	3506.233887	2806.143555	0.004439	0.724063	-1.080432	699.957153	1.000000	6967.483398	0.614117	0.332865	1.616866	00:25
80	3220.140869	3577.505371	2893.383057	0.005646	0.731227	-1.039378	683.952393	1.000000	6991.556152	0.620166	0.329861	1.570222	00:25
81	3208.395508	3488.423340	2788.742188	-0.000503	0.729512	-1.061916	699.554688	1.000000	6825.243652	0.620982	0.335146	1.605686	00:25
82	3191.760254	3466.438232	2781.985352	0.004644	0.736124	-1.030661	684.366333	1.000000	6905.499023	0.620341	0.332895	1.582907	00:25
83	3171.593750	3468.174316	2757.281982	0.013745	0.726387	-1.079011	710.753906	1.000000	6798.083984	0.624436	0.335499	1.625272	00:26
84	3161.748779	3484.867432	2804.370850	0.000396	0.730777	-1.043842	680.357971	1.000000	6938.908203	0.611897	0.331202	1.576850	00:25
85	3145.614746	3414.764160	2711.646484	0.007162	0.726800	-1.078855	703.033142	1.000000	6748.033203	0.614373	0.335700	1.630140	00:25
86	3135.015625	3464.562256	2775.013672	0.002118	0.727726	-1.071037	689.401733	1.000000	6989.595215	0.602924	0.334548	1.618078	00:26
87	3112.888184	3406.750488	2711.639893	0.013885	0.731807	-1.061808	695.017822	1.000000	6812.809570	0.611747	0.336112	1.625439	00:25
88	3104.485596	3372.153564	2665.243896	0.003058	0.727263	-1.091322	706.815735	1.000000	6685.791992	0.606084	0.340014	1.657788	00:25
89	3089.446289	3366.230469	2659.792236	0.007324	0.727721	-1.095376	706.328125	1.000000	6742.983887	0.601475	0.340201	1.673326	00:25
90	3073.190186	3344.774414	2630.106934	0.006065	0.724318	-1.111660	714.586670	1.000000	6634.008789	0.602436	0.340756	1.686546	00:25
91	3060.273926	3337.490479	2629.539795	0.006006	0.727197	-1.101563	707.863281	1.000000	6658.198730	0.598446	0.340994	1.685095	00:26
92	3047.108398	3325.574951	2611.993408	0.016005	0.727472	-1.106509	713.503357	1.000000	6600.878906	0.600146	0.342724	1.696152	00:26
93	3038.662354	3313.505615	2604.016357	0.005596	0.728518	-1.106411	709.403442	1.000000	6611.263672	0.593522	0.343257	1.704652	00:26
94	3017.475830	3303.448486	2587.911377	0.006252	0.727385	-1.113257	715.440002	1.000000	6552.413086	0.596630	0.343977	1.711420	00:25
95	3021.622803	3296.240723	2581.996338	0.005719	0.725646	-1.115477	714.178345	1.000000	6569.918457	0.595726	0.343124	1.704090	00:25
96	3009.111816	3291.238281	2580.051758	0.004037	0.726861	-1.111897	711.094604	1.000000	6562.848633	0.592960	0.343621	1.703392	00:25
97	3000.961182	3284.883545	2570.526123	0.008544	0.726318	-1.115299	714.283325	1.000000	6541.076172	0.594613	0.344068	1.706477	00:25
98	2997.922852	3285.489746	2576.586670	0.007900	0.727089	-1.110541	708.791443	1.000000	6581.392090	0.590692	0.343674	1.701313	00:25
99	2992.362305	3283.334229	2569.966553	0.007610	0.726215	-1.116118	713.273376	1.000000	6540.073730	0.592672	0.344181	1.707559	00:25

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	62175.605469	58420.937500	58348.890625	-0.051949	0.756474	-0.705867	1368.726318	0.047932	61588.308594	1.822992	0.582341	0.659328	00:25
1	49618.242188	39473.527344	39206.046875	-0.043057	0.851662	-0.396860	1403.989502	0.185387	52113.386719	2.003324	0.221583	0.661778	00:25
2	38106.187500	29982.267578	29626.966797	-0.051806	0.686124	-1.142172	902.552917	0.385156	45277.328125	1.447544	0.294982	2.087242	00:25
3	31850.349609	25701.712891	25034.757812	-0.007050	0.893151	-0.651888	1084.826294	0.607671	40963.800781	1.650774	0.426548	1.889337	00:25
4	26937.095703	22336.453125	21880.179688	-0.020213	0.755051	-0.904856	554.679199	0.808860	37525.218750	1.066666	0.338944	1.718830	00:25
5	21971.119141	18084.140625	17574.322266	0.022983	0.661450	-1.064209	531.471375	0.948876	32476.261719	1.040300	0.269217	1.212421	00:25
6	17252.421875	14166.872070	13769.662109	-0.004297	0.650027	-1.045775	395.191437	0.999986	27589.769531	0.842583	0.236650	1.021509	00:25
7	13777.228516	12127.618164	11719.044922	-0.000522	0.590726	-1.253758	407.167450	1.000000	24565.738281	0.784295	0.232350	1.025303	00:25
8	12025.313477	11425.287109	11027.707031	-0.002325	0.577320	-1.306436	396.206085	1.000000	23534.054688	0.737082	0.233305	1.023820	00:25
9	11348.297852	11215.250977	10807.715820	-0.002401	0.565716	-1.354919	405.828369	1.000000	23167.085938	0.733784	0.232911	1.041644	00:25

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	9465.326172	9999.078125	9601.225586	0.007130	0.609639	-1.205733	396.314056	1.000000	21853.699219	0.774112	0.246161	1.020246	00:26
1	8063.734375	7810.102051	7418.045410	-0.010892	0.615129	-1.217685	390.613068	1.000000	17732.515625	0.748079	0.253269	1.108232	00:25
2	7098.869141	6891.984375	6494.851074	0.003510	0.620044	-1.240355	395.655640	1.000000	15703.275391	0.735949	0.262632	1.223647	00:25
3	6485.855469	6270.389648	5868.511719	-0.005693	0.648472	-1.133566	400.792297	1.000000	13933.801758	0.791065	0.267172	1.181391	00:25
4	5972.832520	5844.192871	5451.658691	0.012890	0.644426	-1.175375	391.558289	1.000000	13166.174805	0.748259	0.268520	1.273546	00:25
5	5572.884766	5628.988281	5244.474121	0.007183	0.655677	-1.138401	383.767944	1.000000	12375.596680	0.746120	0.270278	1.270705	00:25
6	5250.625977	5428.687500	5025.041504	0.000618	0.657387	-1.176482	402.747681	1.000000	12075.447266	0.754705	0.281650	1.383355	00:25
7	4992.806152	5028.703125	4640.528320	0.009451	0.677483	-1.110796	387.616364	1.000000	11064.332031	0.747032	0.286410	1.379216	00:25
8	4797.666016	4977.150879	4576.464844	-0.004673	0.662541	-1.180220	399.917969	1.000000	10845.199219	0.740392	0.284689	1.444876	00:25
9	4619.204590	4714.991211	4327.709961	-0.004181	0.676296	-1.131910	386.741913	1.000000	10164.172852	0.729932	0.291930	1.414375	00:25
10	4479.482422	4887.600586	4481.559082	0.000063	0.672485	-1.165117	405.433441	1.000000	10460.187500	0.749355	0.294086	1.469483	00:25
11	4343.376953	4530.287598	4109.966309	0.000223	0.664114	-1.229507	419.752380	1.000000	9527.458008	0.741199	0.301180	1.557173	00:25
12	4236.599609	4525.162598	4090.072266	-0.011065	0.662348	-1.276214	434.732178	1.000000	9375.357422	0.739744	0.307770	1.653390	00:25
13	4161.453613	4408.749512	3994.785400	0.002932	0.679915	-1.200745	413.446655	1.000000	9209.034180	0.729387	0.309570	1.608523	00:26
14	4097.939453	4343.777832	3945.017334	-0.005056	0.684136	-1.152501	398.164246	1.000000	8986.760742	0.727446	0.305452	1.514880	00:25
15	4039.005859	4262.605957	3840.835449	-0.013311	0.676826	-1.218664	421.277039	1.000000	8903.778320	0.734644	0.313449	1.612776	00:25
16	3961.631104	4279.156250	3818.602783	-0.023979	0.654508	-1.387810	460.055573	1.000000	8851.189453	0.717633	0.323141	1.834598	00:25
17	3883.475586	4122.465820	3724.420898	-0.008015	0.687322	-1.149417	397.567566	1.000000	8689.211914	0.723197	0.310432	1.518600	00:25
18	3840.566650	4119.035156	3691.478027	-0.026009	0.669685	-1.277979	427.093781	1.000000	8522.090820	0.709724	0.320246	1.689855	00:25
19	3812.471191	4134.383301	3698.563232	-0.014498	0.668147	-1.309822	435.315338	1.000000	8758.106445	0.705087	0.324950	1.747292	00:25
20	3767.977051	3992.742676	3547.468750	-0.034419	0.659538	-1.350781	444.878937	1.000000	8189.853516	0.704693	0.324789	1.773977	00:25
21	3723.281494	3939.358887	3537.192627	-0.013502	0.689966	-1.189019	401.773529	1.000000	8170.947266	0.695311	0.325168	1.619005	00:25
22	3688.287109	4007.643555	3580.424805	-0.018447	0.674222	-1.281021	426.764069	1.000000	8233.244141	0.699469	0.327571	1.716607	00:26
23	3678.601562	3923.854248	3505.163086	-0.016415	0.674986	-1.260992	418.299408	1.000000	8065.615723	0.694784	0.326394	1.671506	00:25
24	3630.250244	3908.575195	3461.221436	-0.023672	0.666459	-1.361809	446.985535	1.000000	7965.235840	0.690277	0.335831	1.830896	00:25
25	3626.498291	3903.759521	3496.674316	-0.013907	0.686217	-1.209509	406.692993	1.000000	8029.478516	0.693128	0.329922	1.624238	00:25
26	3597.376465	3885.768066	3456.359863	-0.008869	0.673719	-1.306522	428.977020	1.000000	7959.312012	0.683002	0.335279	1.756485	00:25
27	3593.810059	3988.692871	3541.672119	-0.011080	0.669500	-1.339139	446.569672	1.000000	8099.977051	0.701862	0.337833	1.797264	00:25
28	3573.325195	3829.908447	3395.960693	-0.011860	0.675189	-1.295936	433.591003	1.000000	7721.243164	0.699376	0.335587	1.742384	00:26
29	3564.660889	3786.928711	3345.248535	-0.024480	0.668474	-1.339564	441.403625	1.000000	7804.659668	0.690355	0.338523	1.781669	00:25
30	3538.487793	3791.620361	3366.694336	-0.016601	0.676776	-1.271384	424.608124	1.000000	7792.783691	0.696108	0.333885	1.694794	00:25
31	3548.714844	3835.811523	3416.373779	-0.026706	0.682827	-1.228750	419.062561	1.000000	7792.014648	0.708988	0.331578	1.643911	00:25
32	3513.774902	3761.795898	3319.690430	-0.018128	0.669407	-1.317629	441.705902	1.000000	7464.705566	0.707018	0.336299	1.741212	00:25
33	3502.417969	3753.980469	3323.896729	-0.011950	0.676053	-1.294433	429.727722	1.000000	7533.951172	0.689708	0.338480	1.737193	00:26
34	3477.667969	3709.925049	3269.001709	-0.004544	0.668855	-1.335782	440.547546	1.000000	7495.550781	0.690255	0.340251	1.772696	00:25
35	3460.046631	3735.198975	3312.612793	-0.010980	0.679006	-1.271484	422.251831	1.000000	7543.686035	0.685776	0.339557	1.700740	00:25
36	3457.488770	3756.474609	3301.977539	-0.015253	0.663251	-1.366208	454.068909	1.000000	7613.058105	0.702964	0.341805	1.795775	00:25
37	3431.430176	3697.653564	3243.665283	-0.014459	0.669075	-1.351062	453.609192	1.000000	7452.788086	0.705302	0.345873	1.798131	00:25
38	3417.913086	3756.547607	3317.378418	-0.013135	0.669685	-1.324888	438.777130	1.000000	7666.058594	0.692281	0.342241	1.745103	00:25
39	3409.144043	3647.353027	3204.307373	-0.020123	0.668135	-1.331397	442.705261	1.000000	7376.371582	0.697968	0.341519	1.750849	00:25
40	3407.758301	3649.263428	3219.372070	-0.006288	0.682669	-1.265153	429.512146	1.000000	7415.768066	0.700749	0.343829	1.706248	00:25
41	3382.909668	3686.329590	3223.107910	-0.011264	0.663852	-1.383487	462.748718	1.000000	7278.731445	0.707191	0.346626	1.832180	00:25
42	3376.441406	3638.436035	3196.129883	-0.002653	0.672465	-1.340442	441.959717	1.000000	7526.583008	0.681990	0.349581	1.793306	00:25
43	3373.004639	3626.475586	3170.622314	-0.014423	0.665697	-1.368459	455.530701	1.000000	7291.094727	0.698909	0.349142	1.801970	00:25
44	3366.769287	3631.925293	3173.363770	-0.018959	0.659315	-1.397636	458.319580	1.000000	7270.541016	0.691233	0.347236	1.822071	00:25
45	3347.360840	3574.262451	3142.636963	-0.013239	0.668440	-1.326434	431.249786	1.000000	7364.531738	0.674849	0.342823	1.737871	00:25
46	3317.150879	3598.666748	3156.432617	-0.001533	0.668835	-1.350455	441.818359	1.000000	7342.520020	0.678997	0.347351	1.794104	00:25
47	3323.658203	3565.412109	3115.931641	-0.008903	0.667702	-1.356261	449.135681	1.000000	7219.029785	0.692640	0.346469	1.800012	00:25
48	3311.372559	3567.005615	3113.006104	-0.010357	0.668052	-1.343055	453.667999	1.000000	7082.435059	0.711734	0.346697	1.768867	00:25
49	3321.950195	3582.604004	3113.802490	0.000333	0.656901	-1.400587	468.360321	1.000000	7102.209473	0.714338	0.346219	1.814144	00:26
50	3301.511719	3623.174561	3182.668213	0.008003	0.671113	-1.321199	440.130829	1.000000	7262.528320	0.694835	0.346230	1.740687	00:25
51	3285.044678	3552.255371	3106.276123	-0.006173	0.669649	-1.354133	445.727448	1.000000	7177.645996	0.682762	0.350433	1.800857	00:25
52	3286.866455	3675.446045	3236.316650	-0.013157	0.676845	-1.295927	438.723022	1.000000	7318.729980	0.703599	0.346884	1.728112	00:25
53	3291.747803	3667.880127	3209.201904	-0.017041	0.670350	-1.326245	458.201050	1.000000	7180.023926	0.729949	0.348613	1.740342	00:25
54	3284.672852	3593.201660	3139.171631	-0.014269	0.670780	-1.354988	453.624573	1.000000	7211.184570	0.697685	0.352816	1.806872	00:25
55	3267.846924	3536.163086	3085.779541	-0.002470	0.663568	-1.371893	450.007965	1.000000	7252.326660	0.686323	0.348915	1.796228	00:25
56	3250.682617	3564.268555	3101.536377	-0.012916	0.666759	-1.372767	462.398376	1.000000	7097.330566	0.708008	0.353114	1.815818	00:25
57	3254.443604	3547.814453	3098.741943	-0.006862	0.665436	-1.359174	448.660187	1.000000	7157.088379	0.691026	0.348435	1.784791	00:26
58	3260.725586	3530.732178	3079.662109	-0.003880	0.665441	-1.359928	450.669525	1.000000	7183.651855	0.694633	0.348688	1.782578	00:25
59	3236.969482	3484.071289	3034.793945	0.002332	0.670223	-1.346266	448.978607	1.000000	7092.309082	0.694828	0.351385	1.783404	00:25
60	3225.157715	3552.310791	3117.349854	-0.007228	0.672280	-1.309015	434.601807	1.000000	7282.853027	0.690033	0.345595	1.724956	00:25
61	3220.063232	3524.829834	3085.624756	-0.005145	0.671900	-1.325763	438.820679	1.000000	7248.909180	0.686005	0.349257	1.752840	00:26
62	3227.847168	3484.733643	3023.396729	-0.007903	0.657956	-1.388762	461.027283	1.000000	6973.873535	0.704548	0.348345	1.789521	00:26
63	3190.125000	3485.780029	3038.699463	-0.006577	0.668708	-1.349016	446.739380	1.000000	7149.849121	0.689440	0.351118	1.780027	00:25
64	3197.777832	3448.150146	2999.025879	-0.004879	0.659723	-1.374128	448.797913	1.000000	6996.356934	0.686545	0.346901	1.771503	00:25
65	3207.785889	3439.828369	2991.286133	0.004332	0.666682	-1.338493	448.244476	1.000000	6964.046387	0.704034	0.347361	1.744932	00:25
66	3216.496582	3504.475586	3048.498779	0.005608	0.656708	-1.394906	455.563385	1.000000	7071.123535	0.688872	0.348882	1.791947	00:25
67	3205.509766	3455.002930	3015.201904	-0.002899	0.671343	-1.323578	439.482635	1.000000	7103.756348	0.689148	0.350410	1.740655	00:25
68	3190.522461	3437.476318	2982.573242	-0.001924	0.664799	-1.361731	454.539734	1.000000	6959.311523	0.702032	0.350246	1.781555	00:25
69	3179.375000	3434.706055	2972.019043	-0.000864	0.661195	-1.389103	462.347748	1.000000	6931.499023	0.702403	0.351853	1.810729	00:25
70	3154.015137	3450.184814	3001.612061	-0.000603	0.654439	-1.383468	448.301758	1.000000	7242.174316	0.685155	0.344582	1.757106	00:26
71	3109.754395	3367.349121	2874.592285	0.014566	0.634897	-1.472259	492.536652	1.000000	6862.613281	0.737525	0.344089	1.801952	00:26
72	3043.930176	3312.779297	2833.916260	-0.007480	0.644100	-1.434898	478.643097	1.000000	6766.743652	0.723580	0.348020	1.781140	00:25
73	2990.204834	3322.297363	2830.795410	-0.008267	0.642356	-1.456952	491.218353	1.000000	6806.497559	0.736002	0.350011	1.814443	00:25
74	2982.461426	3225.157471	2743.953613	-0.002834	0.637180	-1.477048	481.043793	1.000000	6698.075684	0.704703	0.349975	1.816552	00:25
75	2936.034180	3200.522705	2732.647705	-0.000358	0.648334	-1.434253	467.748627	1.000000	6595.842285	0.693321	0.353818	1.802200	00:25
76	2917.427979	3188.411133	2711.284668	-0.010846	0.649576	-1.430211	476.969299	1.000000	6641.311035	0.715748	0.352910	1.806378	00:26
77	2900.190918	3208.352539	2738.323975	0.001015	0.643907	-1.449119	469.884857	1.000000	6799.942383	0.692973	0.351222	1.807197	00:25
78	2893.122559	3170.847412	2691.765869	-0.006294	0.643069	-1.466224	479.063080	1.000000	6557.207520	0.699490	0.355875	1.826500	00:25
79	2879.305420	3165.599121	2672.646484	-0.009947	0.638078	-1.504574	492.835205	1.000000	6531.899414	0.706808	0.357110	1.871682	00:26
80	2871.717529	3141.771240	2661.048340	0.008516	0.644238	-1.472341	480.623749	1.000000	6581.002441	0.697051	0.357159	1.850578	00:26
81	2841.466553	3114.589111	2620.065918	-0.013839	0.643850	-1.486288	494.445160	1.000000	6409.358398	0.716239	0.360313	1.870289	00:26
82	2827.166260	3109.616699	2619.346680	-0.004788	0.640471	-1.500353	490.193634	1.000000	6416.575684	0.700102	0.360305	1.870650	00:25
83	2806.600342	3085.924561	2605.619385	0.007370	0.646208	-1.465798	480.237457	1.000000	6450.090820	0.697967	0.358896	1.845136	00:26
84	2791.562256	3063.895020	2580.600586	-0.005374	0.645183	-1.463034	483.229340	1.000000	6309.436035	0.707763	0.358645	1.829669	00:25
85	2770.278076	3034.631836	2558.484863	-0.010086	0.647094	-1.460025	476.043915	1.000000	6346.354004	0.691674	0.359322	1.838968	00:25
86	2755.365479	3014.786133	2529.739014	-0.004383	0.644172	-1.489430	484.962555	1.000000	6343.630371	0.692092	0.361628	1.875817	00:26
87	2738.068115	3026.836182	2534.303467	-0.006969	0.645632	-1.497038	492.442505	1.000000	6328.986816	0.700723	0.364204	1.900079	00:25
88	2720.661865	2986.312256	2502.743164	-0.004747	0.646495	-1.490602	483.493469	1.000000	6292.840332	0.683937	0.365157	1.889861	00:25
89	2698.882568	2965.148682	2481.424072	-0.002050	0.646837	-1.491359	483.654999	1.000000	6172.981934	0.683138	0.365614	1.893745	00:25
90	2688.093262	2944.035889	2457.755615	-0.002745	0.643649	-1.513131	486.249237	1.000000	6184.668945	0.675225	0.367282	1.911071	00:25
91	2674.913086	2943.592041	2454.669922	-0.002578	0.644348	-1.508578	488.867065	1.000000	6166.262207	0.683654	0.367843	1.904162	00:25
92	2655.663086	2926.737305	2436.727295	-0.002487	0.646099	-1.506547	489.977570	1.000000	6111.576660	0.685432	0.368832	1.912587	00:26
93	2640.149414	2910.103271	2416.333984	-0.002656	0.642178	-1.537301	493.722076	1.000000	6126.336426	0.674277	0.370405	1.945017	00:25
94	2629.382812	2903.057617	2412.663330	0.002669	0.642730	-1.537003	490.318329	1.000000	6109.676270	0.665981	0.370631	1.949367	00:25
95	2615.815918	2896.590576	2403.968506	-0.000958	0.644573	-1.534148	492.563263	1.000000	6112.826172	0.670643	0.372265	1.953958	00:26
96	2607.981445	2885.615967	2391.821045	-0.003212	0.642997	-1.540947	493.727631	1.000000	6075.826660	0.669907	0.372110	1.956079	00:25
97	2609.968506	2881.287109	2386.099365	-0.002940	0.642660	-1.548294	495.115814	1.000000	6065.745605	0.667622	0.372756	1.968524	00:25
98	2603.308350	2880.835693	2387.826660	-0.004268	0.643977	-1.538081	492.949677	1.000000	6068.658203	0.669138	0.372360	1.957721	00:25
99	2598.858887	2880.998047	2387.606201	-0.004501	0.643797	-1.540458	493.339844	1.000000	6068.282227	0.668343	0.372554	1.961107	00:25

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	58740.539062	55176.195312	54889.167969	0.120437	0.523871	-1.432346	5821.192383	0.047932	60959.058594	2.827253	0.179028	0.830196	00:25
1	45920.312500	32378.673828	32121.812500	-0.037095	0.680219	-0.876800	1361.199707	0.185387	44998.371094	1.317362	0.198534	0.791697	00:25
2	28567.009766	18783.580078	18389.089844	-0.023480	0.572010	-1.268163	1013.169067	0.385156	32817.027344	0.967404	0.211317	0.809108	00:26
3	19321.156250	13977.159180	13471.999023	-0.019137	0.579537	-1.282815	827.127136	0.607671	27876.337891	0.787131	0.226196	0.940032	00:25
4	14469.906250	12062.826172	11523.424805	-0.031412	0.642168	-1.077186	664.454651	0.808860	25497.568359	0.700015	0.236848	0.963814	00:25
5	11711.579102	10398.442383	9849.642578	-0.026665	0.686315	-0.960859	576.726868	0.948876	23399.960938	0.636489	0.249260	1.035778	00:25
6	10033.545898	9363.882812	8784.699219	-0.025187	0.691321	-0.990074	577.364075	0.999986	21912.365234	0.602564	0.263317	1.168671	00:25
7	9010.625977	8700.817383	8128.997070	-0.021991	0.701658	-0.985699	570.353088	1.000000	21030.861328	0.579129	0.276201	1.233390	00:25
8	8411.709961	8434.400391	7861.457031	-0.025626	0.711409	-0.979375	571.588013	1.000000	20555.650391	0.569511	0.285582	1.291906	00:25
9	8147.944824	8266.574219	7697.082031	-0.025811	0.712803	-0.977301	568.101562	1.000000	20270.447266	0.564367	0.286426	1.297384	00:26

epoch	train_loss	valid_loss	l2_b_mean	mu	std	logvar	weighted_kld	kl_weight	l1_b_mean	mu_sd	std_sd	logvar_sd	time
0	7651.367676	7365.542969	6727.640625	-0.001503	0.707371	-0.946704	636.378296	1.000000	18033.966797	0.688343	0.267682	1.189982	00:28
1	6716.003906	6474.808594	5877.452637	-0.036604	0.720139	-0.926342	595.863586	1.000000	16393.494141	0.638700	0.274323	1.235185	00:28
2	5997.029297	5860.405273	5274.698242	-0.018035	0.736100	-0.887804	584.582947	1.000000	15014.263672	0.634403	0.281981	1.247762	00:28
3	5485.274414	5364.577148	4783.366699	-0.004125	0.744810	-0.855897	580.197937	1.000000	13838.269531	0.644898	0.280916	1.228757	00:28
4	5094.323730	5222.379883	4581.991699	0.001645	0.730681	-0.953955	639.444214	1.000000	13211.501953	0.650297	0.294958	1.385436	00:28
5	4805.841309	4832.098145	4243.259277	-0.007232	0.739415	-0.903473	588.059143	1.000000	12172.434570	0.618609	0.291089	1.310845	00:28
6	4583.566406	4695.562012	4069.434326	0.008616	0.726104	-0.973470	625.385925	1.000000	11474.740234	0.621857	0.297524	1.394230	00:28
7	4416.959961	4745.934570	4140.187012	0.001199	0.740462	-0.918071	604.966614	1.000000	11435.058594	0.623564	0.298082	1.354706	00:28
8	4314.148438	4493.190918	3870.912598	-0.005136	0.733318	-0.939494	621.700745	1.000000	10547.795898	0.636232	0.296641	1.357139	00:28
9	4209.107910	4406.982422	3759.320068	-0.001849	0.727758	-1.000237	647.177917	1.000000	10461.577148	0.620679	0.307983	1.468478	00:28
10	4109.696777	4476.614258	3839.443359	-0.009990	0.732419	-0.969451	636.434265	1.000000	10169.422852	0.627681	0.304734	1.423959	00:28
11	4020.404785	4275.003906	3613.874756	-0.010094	0.728018	-1.003290	660.770203	1.000000	9553.526367	0.634482	0.308647	1.477842	00:28
12	3952.284912	4187.662598	3551.754150	0.005761	0.734981	-0.966541	635.446228	1.000000	9338.792969	0.623738	0.309741	1.428058	00:28
13	3900.007324	4116.317871	3462.226562	-0.006851	0.731415	-0.980627	653.731018	1.000000	8986.081055	0.639830	0.308371	1.442627	00:28
14	3851.263916	4204.222656	3555.328857	-0.012392	0.733164	-0.978597	648.536926	1.000000	9004.118164	0.632044	0.310562	1.446320	00:28
15	3785.325684	4007.054443	3348.741943	-0.000152	0.731587	-1.004563	658.063110	1.000000	8655.161133	0.622582	0.315733	1.497758	00:28
16	3767.278564	4183.640625	3517.904785	-0.008906	0.731445	-1.011055	665.384583	1.000000	8616.531250	0.626313	0.316394	1.513834	00:27
17	3758.536621	4103.028320	3443.043213	0.004418	0.725755	-1.007025	659.650574	1.000000	8621.183594	0.632305	0.310023	1.463498	00:28
18	3725.813477	3991.390381	3325.551514	-0.010483	0.731626	-1.013289	665.482483	1.000000	8288.931641	0.623968	0.317578	1.520346	00:28
19	3693.500732	3983.870850	3324.956543	0.013614	0.727927	-1.009194	658.678589	1.000000	8113.777832	0.625404	0.312643	1.483721	00:28
20	3654.010498	3962.879150	3288.293945	0.001136	0.722652	-1.044258	674.351501	1.000000	8213.806641	0.621986	0.315267	1.532625	00:28
21	3647.004395	3983.133789	3338.108398	0.004072	0.739812	-0.957298	644.710266	1.000000	8219.560547	0.635629	0.312068	1.439436	00:28
22	3617.129395	3887.343506	3229.427979	0.008313	0.730873	-1.007894	657.713196	1.000000	7895.292480	0.620299	0.315637	1.502362	00:28
23	3608.985107	3904.537842	3230.343018	0.000425	0.725290	-1.030399	674.001465	1.000000	7845.946289	0.629264	0.315511	1.516837	00:28
24	3591.437744	3859.909424	3193.618164	0.010155	0.734639	-1.009018	666.101379	1.000000	7719.414551	0.623905	0.318807	1.533824	00:28
25	3577.002197	3895.349121	3224.742920	-0.000528	0.729889	-1.021132	670.441528	1.000000	7743.109375	0.625357	0.318466	1.525949	00:28
26	3571.908447	3821.641602	3162.054688	0.000455	0.734451	-1.004335	659.434448	1.000000	7697.803711	0.619632	0.318668	1.517508	00:28
27	3548.191895	3849.494141	3180.026367	-0.004974	0.725175	-1.028136	669.249573	1.000000	7672.394043	0.626161	0.313801	1.516769	00:28
28	3533.383545	4059.682129	3408.733398	-0.000392	0.734781	-0.987317	650.586487	1.000000	8089.685059	0.623778	0.314767	1.479183	00:28
29	3516.743164	3760.632324	3109.681396	0.006230	0.734859	-0.989964	650.809082	1.000000	7584.377441	0.620539	0.317219	1.483018	00:28
30	3506.385498	3800.312500	3123.494629	-0.005099	0.726935	-1.031704	676.580688	1.000000	7519.890625	0.627771	0.318481	1.531833	00:28
31	3484.509277	3836.599365	3150.462402	-0.003075	0.719542	-1.075821	686.004456	1.000000	7661.994141	0.610878	0.322514	1.581861	00:28
32	3495.000732	3849.746582	3175.043701	0.010477	0.730324	-1.023581	674.504395	1.000000	7673.555176	0.626465	0.321434	1.533255	00:28
33	3487.385498	3826.780029	3129.293945	0.018080	0.733950	-1.026333	697.271606	1.000000	7434.991699	0.645661	0.325797	1.565082	00:28
34	3452.620361	3733.065918	3053.726318	0.004488	0.733624	-1.027590	679.199219	1.000000	7412.557617	0.622742	0.326465	1.560631	00:27
35	3455.627930	3729.029297	3067.841309	-0.000455	0.737738	-0.984265	660.988892	1.000000	7427.024414	0.632531	0.321030	1.482692	00:28
36	3448.309326	3716.426270	3055.599121	0.000706	0.732191	-1.013381	660.638794	1.000000	7478.499512	0.614729	0.322410	1.516803	00:28
37	3425.744873	3717.308594	3067.585449	0.009872	0.737455	-0.981283	649.612366	1.000000	7241.781738	0.621098	0.320518	1.472866	00:28
38	3406.716309	3655.440430	2981.873291	0.019556	0.730443	-1.028612	673.444092	1.000000	7264.535156	0.619335	0.324188	1.543146	00:28
39	3397.938721	3687.032959	2996.398438	0.017997	0.725269	-1.059344	690.533447	1.000000	7285.152832	0.619712	0.328644	1.572508	00:28
40	3396.500488	3704.924805	3025.917236	0.004099	0.733460	-1.027411	678.857178	1.000000	7252.134766	0.621408	0.328601	1.556395	00:28
41	3400.028320	3715.416748	3045.153564	0.004757	0.729243	-1.017588	670.118958	1.000000	7224.676270	0.626427	0.322481	1.500674	00:28
42	3389.938232	3709.358887	3035.591797	0.002872	0.730203	-1.026232	673.540894	1.000000	7237.694824	0.620630	0.326542	1.524723	00:28
43	3378.010010	3683.706787	2999.656250	0.010118	0.723173	-1.058739	683.975342	1.000000	7201.193359	0.614366	0.329194	1.549945	00:28
44	3378.157715	3711.697510	3017.652832	0.010882	0.726414	-1.050995	693.858459	1.000000	7120.746094	0.630194	0.327425	1.563918	00:27
45	3371.152588	3661.925781	2982.276611	0.002141	0.728925	-1.028453	679.527405	1.000000	7222.122070	0.628844	0.324787	1.527632	00:28
46	3362.568604	3678.501709	3016.866455	0.002026	0.734364	-1.003709	661.490845	1.000000	7162.108887	0.619610	0.324967	1.503220	00:28
47	3345.114502	3662.004883	3013.256836	0.007719	0.736944	-0.981593	648.579651	1.000000	7179.213867	0.619943	0.321520	1.468639	00:28
48	3339.086670	3722.939941	3025.527588	0.014281	0.727889	-1.059957	697.148499	1.000000	7148.197266	0.623049	0.331727	1.594466	00:28
49	3337.108643	3629.774658	2950.814941	0.008032	0.728444	-1.030823	678.777283	1.000000	7144.182617	0.625660	0.326432	1.524955	00:28
50	3311.899414	3642.291748	2961.773438	0.001066	0.730269	-1.038538	680.423950	1.000000	7062.062500	0.617276	0.330441	1.555513	00:28
51	3320.456543	3581.983154	2891.198242	0.006517	0.728271	-1.043807	690.645813	1.000000	6893.710449	0.627845	0.331142	1.549931	00:28
52	3307.604736	3677.112793	3016.974121	0.007375	0.736529	-0.997537	659.946533	1.000000	7134.888672	0.619345	0.326338	1.500275	00:28
53	3312.864990	3641.685059	2937.004639	-0.004953	0.720087	-1.083988	704.637512	1.000000	6979.736328	0.622097	0.331982	1.590897	00:28
54	3296.139648	3577.402588	2884.481689	0.004521	0.728107	-1.051127	692.779541	1.000000	7009.988770	0.624056	0.332423	1.568000	00:28
55	3282.488037	3654.056396	2947.325928	0.014008	0.723545	-1.081236	706.626526	1.000000	7055.623535	0.621703	0.333729	1.613418	00:28
56	3289.469238	3602.896240	2920.512939	0.006522	0.728315	-1.034476	682.367432	1.000000	6979.327637	0.626334	0.328607	1.531155	00:28
57	3292.510498	3604.759033	2907.411865	0.001527	0.724153	-1.068953	697.232239	1.000000	7029.240234	0.619716	0.333099	1.584040	00:28
58	3279.932373	3631.697266	2934.322021	0.013448	0.722786	-1.074533	697.274597	1.000000	6931.425781	0.616444	0.333294	1.585446	00:28
59	3263.800537	3766.281738	3084.604980	0.010203	0.727060	-1.042177	681.429016	1.000000	7250.323730	0.619173	0.330562	1.536164	00:28
60	3247.138428	3614.538818	2906.910400	0.012615	0.722017	-1.082184	707.588440	1.000000	6928.161621	0.622702	0.336169	1.592687	00:28
61	3261.955566	3540.338135	2835.608398	0.003212	0.724194	-1.073995	704.604065	1.000000	6790.403320	0.623425	0.335638	1.589854	00:27
62	3256.552002	3535.280029	2826.410645	0.001747	0.721265	-1.087617	708.741272	1.000000	6755.615723	0.621381	0.335210	1.605481	00:28
63	3250.792480	3548.330078	2842.145508	0.018147	0.726087	-1.072173	706.068848	1.000000	6790.458984	0.624347	0.335631	1.605922	00:28
64	3261.629150	3514.951660	2814.265869	0.005941	0.722353	-1.069727	700.618652	1.000000	6820.963379	0.625080	0.333472	1.567700	00:28
65	3241.566650	3608.032715	2904.027832	0.006998	0.723039	-1.076644	703.917480	1.000000	6986.800781	0.622058	0.335172	1.589766	00:28
66	3224.029053	3570.575195	2885.435303	0.016744	0.725186	-1.057620	685.048767	1.000000	6912.272949	0.612871	0.331627	1.565044	00:28
67	3251.290039	3583.719482	2884.775635	0.011005	0.723794	-1.060478	698.897400	1.000000	6890.466797	0.629185	0.332542	1.557823	00:28
68	3238.050049	3528.008789	2828.175049	0.019224	0.726878	-1.064695	699.730469	1.000000	6830.160645	0.621388	0.335719	1.589031	00:28
69	3223.922852	3530.111084	2814.860352	0.006388	0.723904	-1.079526	715.156250	1.000000	6741.392090	0.631759	0.336924	1.602475	00:28
70	3216.445068	3500.061523	2782.776855	0.008921	0.720420	-1.104010	717.167419	1.000000	6705.318359	0.617773	0.338750	1.635759	00:29
71	3207.145996	3521.456787	2817.231201	0.005418	0.720238	-1.086549	704.176575	1.000000	6791.708496	0.616982	0.336408	1.589192	00:28
72	3204.608154	3531.666260	2808.478760	0.009166	0.717051	-1.115761	723.127869	1.000000	6687.869629	0.620199	0.337613	1.643584	00:28
73	3213.811523	3598.167236	2889.819580	0.010586	0.726090	-1.083778	708.206543	1.000000	6976.821777	0.615494	0.339968	1.628908	00:28
74	3203.632324	3670.336182	2951.901855	0.005238	0.719610	-1.084969	718.238342	1.000000	6905.762695	0.637379	0.334413	1.585946	00:28
75	3207.873535	3579.372803	2876.421875	0.006835	0.723253	-1.067832	702.820068	1.000000	6790.896484	0.627843	0.334451	1.570162	00:28
76	3187.589355	3531.233643	2829.830811	0.013975	0.722816	-1.077619	701.346497	1.000000	6883.243164	0.618706	0.334157	1.596420	00:28
77	3181.636719	3545.495117	2852.156006	0.002698	0.731067	-1.049801	693.244995	1.000000	6794.460449	0.620472	0.335831	1.587278	00:28
78	3177.850098	3536.807861	2822.090088	0.005285	0.719301	-1.111130	714.644043	1.000000	6781.951172	0.608447	0.341650	1.637716	00:28
79	3180.280518	3572.795410	2873.986572	0.006143	0.722267	-1.085594	698.616516	1.000000	6843.494141	0.607717	0.337304	1.603625	00:28
80	3168.005371	3535.424561	2833.231201	0.019834	0.726783	-1.064308	702.152222	1.000000	6706.426758	0.623918	0.337304	1.585141	00:28
81	3152.529297	3446.023438	2734.945557	0.008887	0.723976	-1.088619	710.974243	1.000000	6700.422852	0.617348	0.340108	1.623608	00:28
82	3135.922119	3434.467773	2714.553223	0.012614	0.720932	-1.108265	719.850525	1.000000	6591.262695	0.614366	0.343730	1.639695	00:27
83	3113.646729	3466.451172	2765.960205	0.003383	0.726390	-1.076803	700.406921	1.000000	6657.192871	0.610935	0.339992	1.610990	00:28
84	3098.293701	3459.226318	2755.147705	0.016013	0.724201	-1.087780	703.987000	1.000000	6683.789551	0.607959	0.341545	1.616467	00:28
85	3087.032959	3389.620605	2680.396240	0.007750	0.721710	-1.108647	709.155762	1.000000	6485.849121	0.599022	0.344326	1.646617	00:28
86	3070.500977	3382.860840	2672.243164	0.014384	0.723655	-1.103893	710.518250	1.000000	6548.608398	0.602184	0.344518	1.650620	00:28
87	3055.568359	3381.640869	2666.888916	0.006845	0.725388	-1.102032	714.671509	1.000000	6474.822754	0.606631	0.345363	1.659149	00:28
88	3045.261719	3402.307373	2680.911377	0.006631	0.723334	-1.119401	721.316833	1.000000	6557.485352	0.602261	0.347152	1.684888	00:28
89	3029.100830	3327.909180	2613.521240	0.009250	0.723729	-1.116796	714.319580	1.000000	6478.024902	0.594214	0.348252	1.677276	00:28
90	3009.680908	3317.802979	2615.427002	0.011575	0.728394	-1.091922	702.291748	1.000000	6491.918945	0.594770	0.345983	1.656690	00:28
91	2996.286865	3309.975342	2594.326660	0.006130	0.726961	-1.113291	715.553467	1.000000	6424.102539	0.593813	0.349996	1.692982	00:28
92	2977.406250	3292.775391	2571.700195	0.013427	0.725092	-1.122516	721.000305	1.000000	6330.194824	0.594845	0.350828	1.698746	00:27
93	2972.783447	3268.155273	2544.673096	0.008637	0.725328	-1.129471	723.458557	1.000000	6335.770508	0.591184	0.352287	1.717149	00:28
94	2970.029785	3270.115967	2551.228760	0.009532	0.724969	-1.130088	718.871765	1.000000	6373.247559	0.585079	0.352135	1.716223	00:28
95	2954.962891	3261.302979	2542.979004	0.008221	0.725357	-1.124736	718.260071	1.000000	6304.031250	0.588493	0.351823	1.705143	00:28
96	2950.425049	3252.043213	2532.653076	0.010782	0.725520	-1.130057	719.378235	1.000000	6317.508301	0.584415	0.353175	1.717738	00:28
97	2943.955322	3251.385254	2532.198486	0.008705	0.726147	-1.125400	719.156494	1.000000	6292.142090	0.587820	0.352392	1.712947	00:28
98	2944.449463	3245.879883	2527.619141	0.008804	0.725314	-1.130287	718.228455	1.000000	6309.519043	0.583112	0.352947	1.717500	00:28
99	2937.538330	3244.074463	2529.292725	0.008952	0.725937	-1.125587	714.744995	1.000000	6312.990234	0.581985	0.352475	1.711296	00:28

epoch	train_loss	valid_loss	l2_b_mean	mmd	mu	std	logvar	l1_b_mean	mu_sd	l1_latent_reg	weighted_kld	logvar_sd	time
0	35639.734375	33119.378906	32758.919922	354.753448	-0.645665	2.168537	1.389641	45340.976562	5.047215	5219.156250	18799.441406	0.807266	00:24
1	22419.580078	15466.881836	15300.407227	162.598511	-0.390403	2.085497	1.324767	30500.722656	3.677496	3760.603516	10589.691406	0.736531	00:23
2	14881.570312	10914.019531	10817.428711	94.298996	-0.218614	1.718532	0.952760	24908.886719	2.857116	2908.460693	6201.050781	0.693021	00:23
3	10775.079102	8509.122070	8455.727539	51.379539	-0.116926	1.412444	0.578432	21353.609375	2.194111	2228.759521	3532.319580	0.640258	00:24
4	8417.246094	7147.596680	7119.229980	26.634216	-0.072432	1.150160	0.179836	18936.121094	1.669925	1692.649536	1983.132935	0.602620	00:24
5	6787.373047	6221.455566	6203.451660	16.419044	-0.051861	0.946574	-0.200824	17108.943359	1.339317	1349.057983	1275.688110	0.569902	00:23
6	5708.291504	5401.690430	5387.585449	12.589787	-0.049178	0.882773	-0.334380	15516.946289	1.213070	1216.506592	1068.951294	0.555498	00:23
7	4967.996582	4810.905273	4799.205566	10.367170	-0.035553	0.831857	-0.451990	14039.386719	1.123102	1120.465332	946.840881	0.552006	00:24
8	4480.203613	4512.324219	4501.353516	9.820558	-0.032393	0.812645	-0.496762	13347.663086	1.094568	1089.507446	911.750488	0.546974	00:23
9	4122.484863	4195.836426	4185.203613	9.572933	-0.030994	0.816746	-0.486494	12404.958984	1.091210	1083.963379	904.418335	0.547605	00:24
10	3843.719238	3955.936768	3945.632324	9.421382	-0.027061	0.806554	-0.510669	11746.567383	1.079526	1070.157715	891.319214	0.544316	00:24
11	3627.255371	3750.461670	3739.786133	9.974444	-0.035787	0.810802	-0.502045	11090.406250	1.095940	1082.943848	915.384888	0.550083	00:24
12	3453.144775	3555.236572	3544.947998	9.497383	-0.033756	0.805314	-0.512734	10532.422852	1.081325	1065.357910	893.720581	0.540843	00:24
13	3317.225586	3445.394287	3435.099854	9.603128	-0.026698	0.807787	-0.506707	10088.870117	1.086125	1070.625977	898.874268	0.540827	00:23
14	3207.483887	3477.685303	3467.043945	9.804630	-0.036560	0.803424	-0.516729	10080.664062	1.088416	1068.231079	904.268982	0.537274	00:23
15	3097.683838	3296.834473	3286.019287	10.156882	-0.024965	0.809974	-0.501416	9402.141602	1.100944	1077.333496	919.126099	0.541100	00:24
16	3026.498535	3239.862305	3229.419434	9.751334	-0.027817	0.814940	-0.486714	9327.415039	1.095807	1070.757568	907.116394	0.533075	00:23
17	2942.689209	3145.737793	3134.896729	10.162513	-0.031712	0.819668	-0.476477	8857.916992	1.109087	1079.275024	925.773315	0.536413	00:24
18	2910.235596	3092.333984	3081.208252	10.555488	-0.016682	0.829229	-0.454906	8585.517578	1.126567	1096.386475	948.013977	0.542863	00:23
19	2832.958252	3017.066650	3006.289307	10.245089	-0.018359	0.826162	-0.462675	8570.823242	1.114929	1082.901733	933.252991	0.543466	00:23
20	2772.829590	3046.259277	3036.564209	9.047776	-0.024695	0.789895	-0.547899	8529.547852	1.059283	1029.493530	866.595886	0.529360	00:24
21	2753.147461	2986.807861	2975.513428	10.669466	-0.013018	0.844861	-0.416567	8073.350098	1.141360	1104.890015	962.535461	0.538116	00:23
22	2712.455566	2906.891602	2896.313965	10.020215	-0.016617	0.819964	-0.475952	8024.366211	1.107157	1071.144531	922.119934	0.536537	00:23
23	2677.711914	3020.316895	3009.638916	10.101330	-0.021751	0.810849	-0.496884	8298.047852	1.099663	1062.236694	914.141846	0.532246	00:23
24	2626.546875	2862.383057	2852.536377	9.309927	-0.015195	0.803757	-0.513902	8120.997070	1.077071	1039.144897	884.831543	0.530815	00:23
25	2575.743164	2870.408691	2860.929199	9.051567	-0.026579	0.797159	-0.528016	8179.134766	1.062291	1025.824219	865.369934	0.521008	00:23
26	2480.682617	2658.409668	2648.303711	9.851988	-0.019407	0.812116	-0.495792	7458.644043	1.096348	1055.604858	911.445984	0.536942	00:24
27	2367.523193	2564.386475	2555.031738	9.078519	-0.010155	0.785778	-0.558931	7359.857910	1.055575	1014.959229	864.135681	0.527093	00:23
28	2287.362793	2577.752686	2568.370605	8.995213	-0.011609	0.781813	-0.570015	7215.798340	1.051848	1011.241882	862.347351	0.529978	00:24
29	2229.250488	2449.017822	2440.301514	8.443208	-0.008403	0.766377	-0.607935	7179.539551	1.025925	984.454834	834.755188	0.523161	00:23
30	2194.789062	2457.297119	2448.329590	8.684980	-0.008512	0.759338	-0.627476	7038.735840	1.025120	981.841309	838.818848	0.527536	00:24
31	2152.750732	2365.702148	2356.650146	8.731876	-0.015857	0.768805	-0.602370	6879.909668	1.036128	993.249329	847.809448	0.525540	00:23
32	2104.733643	2339.498535	2330.842041	8.281164	-0.006147	0.755411	-0.635174	6761.827637	1.013497	971.661621	823.330444	0.517066	00:24
33	2079.943604	2352.695068	2343.459961	8.989347	0.000814	0.767590	-0.606251	6705.492676	1.039513	994.690674	854.078735	0.525999	00:23
34	2057.652100	2289.373535	2281.190186	8.049372	-0.003593	0.754654	-0.636665	6767.381348	1.005222	961.257935	812.971985	0.514499	00:23
35	2048.321533	2333.577881	2325.027100	8.287884	-0.004861	0.745774	-0.659917	6811.984375	1.004853	959.555969	817.796509	0.513304	00:23
36	2008.904785	2287.277344	2278.947998	8.144099	-0.007205	0.744801	-0.662207	6504.483887	1.000480	952.984863	812.662354	0.512287	00:24
37	1978.226929	2199.820557	2191.496582	8.082468	-0.014684	0.740286	-0.675295	6599.623535	0.994912	947.844360	809.869019	0.514765	00:24
38	1965.509399	2354.071045	2345.678467	8.060910	-0.011115	0.751032	-0.645638	6712.160156	1.002705	954.138611	811.483521	0.512083	00:23
39	1944.204224	2543.402832	2531.188232	11.698755	-0.002674	0.802894	-0.519100	6909.925781	1.131840	1088.012329	967.922241	0.531437	00:23

epoch	train_loss	valid_loss	l2_b_mean	mmd	mu	std	logvar	l1_b_mean	mu_sd	l1_latent_reg	weighted_kld	logvar_sd	time
0	44869.117188	38630.105469	38172.390625	452.347382	0.363450	2.003438	1.154238	49319.636719	4.148453	8080.807617	26520.181641	0.966020	00:23
1	28217.714844	19543.214844	19365.693359	173.174393	0.211772	1.590892	0.723334	34397.785156	2.722190	5338.421875	11395.580078	0.892636	00:23
2	18315.355469	13255.430664	13121.753906	130.631409	0.219888	1.532194	0.665691	27403.474609	2.465842	4917.343750	9348.974609	0.835872	00:23
3	12825.623047	9738.818359	9650.188477	86.012886	0.174977	1.367238	0.465474	22967.119141	2.081707	4158.813965	6535.408691	0.760577	00:24
4	9639.348633	7791.971191	7737.542480	52.770176	0.111142	1.187918	0.203920	20011.353516	1.711145	3402.236816	4313.964844	0.716786	00:24
5	7687.958496	6969.908203	6936.609375	31.604336	0.071600	1.017000	-0.096666	18651.474609	1.397718	2771.013672	2869.612793	0.689790	00:24
6	6420.611328	5956.696289	5929.208008	25.928036	0.055198	0.933785	-0.259582	16708.939453	1.273471	2517.689209	2402.669678	0.671029	00:24
7	5523.093750	5197.247559	5173.945801	21.902893	0.056021	0.890996	-0.346857	14908.278320	1.195464	2358.052979	2146.462646	0.652390	00:23
8	4918.751465	4908.517578	4887.394043	19.962851	0.048813	0.845242	-0.446389	13966.420898	1.135524	2239.118164	1974.152588	0.633212	00:24
9	4473.037598	4544.405762	4524.976074	18.338888	0.045379	0.825765	-0.490280	13264.149414	1.100679	2152.622803	1881.080322	0.627626	00:23
10	4142.837402	4177.356934	4157.032715	19.363203	0.038059	0.848681	-0.432753	12074.405273	1.130283	2209.923340	1944.031250	0.618873	00:24
11	3917.388916	3997.903320	3977.184814	19.701015	0.057482	0.848224	-0.430466	11531.174805	1.133727	2215.214111	1949.137207	0.608181	00:24
12	3699.811768	3793.642334	3773.311523	19.624884	0.058232	0.857292	-0.405696	10980.250000	1.137904	2227.622314	1948.683960	0.595239	00:24
13	3548.590576	3717.894043	3695.793945	21.325510	0.051739	0.856475	-0.406966	10586.673828	1.154981	2252.016113	1997.359985	0.593199	00:24
14	3392.223633	3564.068848	3543.844238	19.469976	0.048081	0.859627	-0.396350	10151.842773	1.136936	2212.456299	1934.764526	0.582036	00:24
15	3294.741943	3458.957520	3437.580811	20.652731	0.035036	0.864861	-0.385641	9724.040039	1.155857	2242.773926	1989.750732	0.587111	00:24
16	3188.714355	3471.833740	3450.045410	21.072681	0.061831	0.877749	-0.353019	9386.150391	1.169560	2276.136963	2020.442383	0.577378	00:24
17	3111.545898	3386.387207	3364.288574	21.481207	0.057307	0.876575	-0.355280	9266.531250	1.172527	2275.385742	2028.340576	0.576368	00:24
18	3067.808838	3286.744629	3263.543457	22.496641	0.058035	0.901485	-0.299061	8925.688477	1.206159	2334.083984	2121.989258	0.577001	00:23
19	3007.293945	3189.976807	3167.430908	21.941383	0.049392	0.884795	-0.334096	8704.542969	1.184379	2293.989990	2053.986328	0.570027	00:23
20	2945.777100	3182.058105	3159.409668	22.111326	0.049436	0.907009	-0.285508	8585.974609	1.205574	2330.225830	2113.958740	0.571989	00:23
21	2895.951416	3211.220459	3188.865967	21.689819	0.046961	0.886839	-0.327073	8588.288086	1.182075	2284.830322	2039.987183	0.562882	00:24
22	2849.264160	3051.117188	3028.877930	21.914141	0.062428	0.886953	-0.327097	8263.973633	1.183866	2291.922607	2047.876343	0.563708	00:24
23	2808.139648	3054.230713	3032.758301	20.943661	0.052907	0.880357	-0.340083	8256.331055	1.169980	2262.256104	2002.924072	0.556725	00:24
24	2773.731201	3043.178711	3019.842285	22.779768	0.056994	0.902545	-0.291591	8085.221680	1.206399	2338.058105	2110.115967	0.558648	00:23
25	2742.290527	2963.630859	2941.413818	21.570345	0.058292	0.876817	-0.347950	7992.745117	1.172185	2268.051758	2011.426392	0.555742	00:23
26	2719.287354	2948.922852	2925.617676	22.668314	0.061004	0.901012	-0.293574	7791.686523	1.204079	2328.265869	2100.627441	0.555158	00:23
27	2679.970215	2916.285645	2894.107422	21.886332	0.058098	0.897683	-0.300988	7793.752441	1.191654	2299.196289	2063.407471	0.554300	00:23
28	2657.288818	2888.732910	2867.198486	21.124582	0.054443	0.885014	-0.327630	7721.035645	1.174288	2261.996826	2010.303223	0.548887	00:23
29	2637.341553	2875.486816	2852.899658	22.185432	0.052402	0.899004	-0.297901	7635.975098	1.196017	2309.848877	2075.057373	0.552954	00:24
30	2611.630371	2916.487549	2893.916016	22.165520	0.060199	0.892029	-0.311832	7624.857910	1.190608	2294.882568	2058.446533	0.548067	00:23
31	2601.701660	2958.729736	2937.280029	21.088247	0.052859	0.875972	-0.348007	7780.184570	1.164899	2237.513916	1986.447510	0.547276	00:24
32	2560.690186	2890.773193	2868.005615	22.291035	0.053579	0.893000	-0.311326	7511.469238	1.193920	2301.701660	2071.535645	0.552681	00:24
33	2536.779785	2881.475586	2859.619873	21.348116	0.060162	0.896214	-0.303677	7471.465332	1.185748	2284.387695	2045.750488	0.550699	00:24
34	2520.879150	2919.730957	2896.383057	22.488171	0.056736	0.900711	-0.294199	7453.818848	1.202283	2313.757324	2095.693848	0.553407	00:23
35	2495.342773	2774.852539	2752.407227	21.960470	0.051357	0.895464	-0.304288	7264.982422	1.192231	2286.324463	2062.578125	0.546508	00:23
36	2484.418701	2731.136230	2710.085938	20.814085	0.050483	0.880643	-0.337369	7257.747559	1.166030	2236.843018	1987.916504	0.546926	00:23
37	2467.477051	2689.622559	2667.826416	21.342794	0.057822	0.896463	-0.301654	7114.496094	1.185970	2275.238037	2043.145142	0.544947	00:23
38	2457.845703	2991.763428	2970.408203	21.072306	0.052153	0.889041	-0.318461	7656.958008	1.176745	2257.115234	2017.733765	0.545500	00:24
39	2322.232178	2856.982422	2834.868896	21.692192	0.040446	0.881323	-0.335423	7360.048828	1.178361	2268.504395	2023.965210	0.540828	00:24

epoch	train_loss	valid_loss	l2_b_mean	mmd	mu	std	logvar	l1_b_mean	mu_sd	l1_latent_reg	weighted_kld	logvar_sd	time
0	43863.750000	38415.714844	38403.582031	4.531198	-0.005897	0.299329	-2.509599	52373.148438	0.443983	215.868622	578.306458	0.593813	00:24
1	29051.519531	25556.123047	25545.445312	6.811570	-0.004425	0.374405	-2.064293	41032.382812	0.560199	277.017273	490.831116	0.606257	00:23
2	20931.611328	19844.193359	19834.179688	7.637002	-0.004937	0.397987	-1.930601	35639.593750	0.592585	290.151154	466.145905	0.562413	00:24
3	14663.121094	12426.390625	12416.396484	8.592687	-0.002675	0.421610	-1.805782	27062.740234	0.626879	308.991547	445.806946	0.527288	00:24
4	10193.139648	9417.468750	9406.019531	10.110537	-0.011966	0.455588	-1.648575	22221.794922	0.682492	335.222321	429.319397	0.516746	00:23
5	7524.279785	6364.255859	6352.415039	10.702779	0.011146	0.471224	-1.580377	17584.408203	0.701387	342.728180	421.015381	0.519270	00:23
6	6115.612305	5585.450684	5572.469727	11.632233	-0.000796	0.494548	-1.479531	15968.841797	0.733156	358.945038	410.607941	0.503936	00:23
7	5255.900879	5021.166016	5008.228516	11.662443	-0.007470	0.507035	-1.426110	14532.230469	0.745946	364.898315	403.718201	0.487873	00:23
8	4705.863281	4692.226562	4679.104004	11.865071	0.000847	0.522057	-1.364778	13638.712891	0.759972	370.586456	396.001831	0.477372	00:24
9	4339.275391	4509.605469	4495.935059	12.448554	-0.006662	0.537097	-1.307204	13147.251953	0.781810	380.401398	393.854126	0.474533	00:23
10	4063.217041	4339.301758	4325.839355	12.518625	-0.000959	0.539867	-1.295752	12419.667969	0.782707	379.506470	391.445526	0.469356	00:24
11	3844.226074	3915.171143	3901.546631	12.703267	-0.002186	0.557496	-1.232466	11357.519531	0.798980	386.335205	386.182617	0.474052	00:24
12	3663.054199	4350.450684	4336.080566	13.246813	-0.008105	0.564001	-1.208974	11926.161133	0.812054	393.216003	387.989990	0.474185	00:24
13	3521.080322	3717.507324	3703.758057	12.813422	-0.001015	0.569932	-1.187693	10647.582031	0.811043	391.481812	382.930420	0.473800	00:23
14	3417.034912	3798.777100	3783.877441	14.170590	-0.016727	0.591292	-1.114504	10922.300781	0.846533	408.519867	386.855652	0.475482	00:24
15	3295.059814	3525.010742	3510.413818	13.814376	-0.007868	0.580232	-1.151974	10026.403320	0.834055	402.132599	387.572205	0.472764	00:24
16	3211.870850	3535.393066	3520.083984	14.482000	-0.007838	0.606115	-1.062313	9636.927734	0.860866	415.887024	384.029480	0.462664	00:24
17	3143.329590	3317.885498	3303.194824	14.088883	-0.001076	0.601462	-1.077590	9092.801758	0.851771	410.015045	381.872803	0.463364	00:24
18	3076.614990	3438.985596	3424.058838	14.395399	0.000178	0.608181	-1.056857	9169.375000	0.863004	413.457825	384.296448	0.470794	00:24
19	3009.324219	3417.989014	3403.146240	14.122858	-0.007747	0.606345	-1.062125	9251.507812	0.858551	413.128448	382.520508	0.467891	00:23
20	2935.841309	3159.706299	3144.865479	14.208220	-0.003280	0.608992	-1.053701	8666.197266	0.862335	413.299194	383.190063	0.468349	00:23
21	2889.335449	3118.774902	3104.639648	13.635227	0.006028	0.614202	-1.034251	8542.797852	0.857498	410.562469	376.163757	0.459089	00:23
22	2904.316650	3182.259766	3166.236572	15.280206	-0.009078	0.650401	-0.921584	8301.858398	0.909851	435.520782	386.079926	0.463378	00:23
23	2842.148438	3260.038330	3245.093750	14.698691	-0.003224	0.629414	-0.985648	8522.114258	0.881078	422.249756	380.361420	0.460035	00:24
24	2805.152588	3302.236084	3285.666748	16.182764	-0.007153	0.645778	-0.935025	8513.980469	0.916546	439.027618	391.966736	0.460427	00:24
25	2785.724121	3376.423096	3359.880615	15.800976	-0.001768	0.657435	-0.901159	8544.414062	0.920507	438.538971	389.170227	0.466704	00:24
26	2762.451416	2964.311523	2949.358643	14.561893	-0.000443	0.647257	-0.929403	8000.426758	0.894137	427.106110	377.525421	0.458053	00:23
27	2724.360596	2976.501221	2960.346680	15.885023	-0.002973	0.651166	-0.919243	7846.898926	0.915923	437.352905	388.857391	0.463826	00:24
28	2703.101807	2907.522461	2892.239502	14.919702	-0.004438	0.654061	-0.908257	7749.744629	0.905100	431.247742	380.027710	0.455881	00:23
29	2670.739502	2896.980957	2881.499268	14.934103	-0.003205	0.652672	-0.913074	7585.269531	0.907845	433.303772	382.666962	0.456568	00:23
30	2657.177246	2908.962158	2893.283447	15.238147	0.002911	0.667215	-0.869191	7627.063477	0.921041	438.859680	382.833466	0.459951	00:24
31	2632.270264	2971.827881	2956.658691	14.708002	0.000174	0.651452	-0.916749	7723.069336	0.901899	430.017609	379.759186	0.459874	00:23
32	2633.249023	2896.732422	2881.563477	14.930392	0.000247	0.654663	-0.906587	7833.086426	0.906260	431.945465	380.294464	0.458233	00:23
33	2599.547119	2908.443604	2893.063965	14.918993	0.005476	0.661886	-0.884253	7560.501953	0.911457	433.857971	379.544250	0.454524	00:24
34	2592.372803	2866.823975	2851.997314	13.981042	0.007245	0.658070	-0.894802	7673.434082	0.896489	426.589722	372.266479	0.451590	00:24
35	2572.281494	2804.459961	2789.447754	14.607452	0.005032	0.661891	-0.885020	7414.482910	0.908774	433.066833	378.309479	0.458576	00:24
36	2559.514160	2804.730713	2789.944824	14.492640	0.009472	0.655917	-0.902291	7479.042969	0.900253	428.058014	376.270325	0.454680	00:24
37	2549.586182	2803.934570	2788.795410	14.567461	0.001700	0.673275	-0.849742	7306.768555	0.917193	435.740265	377.050690	0.453040	00:24
38	2542.187500	2838.276611	2822.072754	15.779968	0.002494	0.678474	-0.836513	7458.104004	0.938841	446.885040	388.480835	0.460749	00:23
39	2528.304443	2727.795166	2712.283447	15.043669	0.010385	0.680495	-0.828586	7119.888672	0.929208	441.239685	380.757141	0.454719	00:23

epoch	train_loss	valid_loss	l2_b_mean	mmd	mu	std	logvar	l1_b_mean	mu_sd	l1_latent_reg	weighted_kld	logvar_sd	time
0	40959.515625	37905.636719	37891.761719	8.208526	-0.024008	0.321359	-2.330088	51717.753906	0.548548	277.020477	558.442444	0.462210	00:23
1	28461.808594	31310.380859	31296.017578	11.071305	-0.001257	0.356824	-2.144540	45347.582031	0.634957	318.794250	540.526062	0.561912	00:24
2	20569.023438	24270.050781	24254.761719	12.098309	-0.021030	0.391993	-1.954344	39474.792969	0.675133	337.234344	506.135071	0.548326	00:23
3	14445.319336	15024.644531	15008.929688	13.079365	-0.020070	0.419365	-1.815197	30104.261719	0.710429	353.417725	484.618500	0.530492	00:24
4	10298.541016	9405.922852	9392.183594	12.310252	-0.012689	0.445155	-1.688505	22990.738281	0.714922	355.908539	453.056366	0.504430	00:24
5	7728.354980	6665.394531	6650.482422	13.403973	-0.013079	0.452304	-1.656670	18389.359375	0.737011	364.198914	455.659149	0.502500	00:23
6	6190.182129	5603.939941	5589.669434	12.933229	-0.013793	0.479955	-1.534696	15767.240234	0.748866	369.001404	430.846649	0.491857	00:23
7	5236.472656	5212.584961	5196.565430	14.607766	-0.016758	0.491669	-1.485687	14362.958984	0.785005	385.483612	437.185181	0.486615	00:23
8	4617.360840	4473.277344	4458.379395	14.115689	-0.030596	0.498461	-1.451891	12745.385742	0.778319	385.171143	424.746368	0.459398	00:23
9	4208.974609	4216.414551	4200.229004	15.131713	-0.027884	0.513088	-1.392592	11635.668945	0.803424	394.603180	423.435181	0.455078	00:23
10	3901.279785	4004.926025	3988.525146	15.441641	-0.022382	0.523603	-1.348659	10846.424805	0.816376	402.754547	419.191315	0.437780	00:24
11	3670.582275	3740.616455	3724.684570	15.171966	-0.032123	0.524357	-1.344947	10166.298828	0.812432	399.097137	416.266876	0.438142	00:24
12	3491.620117	3667.437744	3651.875488	14.801250	-0.026936	0.534644	-1.305375	9781.682617	0.815828	401.885956	408.844971	0.435637	00:24
13	3373.058594	3597.774658	3581.469238	15.714341	-0.029027	0.541989	-1.278454	9533.083984	0.833115	409.734467	412.325806	0.437646	00:23
14	3268.388672	3494.202637	3478.181396	15.434155	-0.029170	0.551247	-1.245216	9079.128906	0.836138	408.363190	406.691254	0.440574	00:24
15	3192.942139	3778.173584	3761.590820	15.698897	-0.017824	0.561030	-1.210492	9206.311523	0.848363	414.858185	405.788605	0.443565	00:24
16	3132.410400	3364.331055	3348.152100	15.786729	-0.019340	0.569641	-1.177485	8492.832031	0.853677	417.495789	401.234131	0.430003	00:23
17	3073.229492	3397.922852	3381.511230	15.826709	-0.029253	0.571797	-1.171749	8525.550781	0.855620	416.931549	401.589661	0.439019	00:24
18	3024.521240	3429.331299	3412.252441	16.369137	-0.021574	0.587442	-1.115198	8408.209961	0.877177	429.384705	401.066406	0.427119	00:23
19	2968.729004	3244.765137	3229.118896	15.019815	-0.023893	0.582480	-1.132304	8225.012695	0.854739	416.707214	391.953735	0.428889	00:23
20	2925.205078	3132.208984	3115.920166	16.067533	-0.026519	0.596821	-1.084884	7874.996582	0.875232	426.971710	394.178131	0.432377	00:24
21	2889.247314	3118.166748	3102.901123	15.002897	-0.029995	0.585165	-1.121761	7893.246582	0.851289	415.738831	387.821686	0.424242	00:24
22	2845.302734	3130.704346	3114.590576	15.593694	-0.020829	0.592054	-1.099054	7766.731934	0.867235	421.982574	392.017334	0.426566	00:24
23	2807.441162	3189.631836	3173.846924	14.906292	-0.021064	0.596690	-1.083462	7741.119629	0.862720	421.786804	386.648254	0.426360	00:23
24	2784.897949	3024.525879	3008.107178	15.926530	-0.020563	0.606408	-1.051424	7504.205566	0.881879	429.748260	391.035858	0.426201	00:23
25	2770.114746	3029.304199	3013.619873	15.341253	-0.023551	0.608746	-1.042673	7513.146973	0.875445	425.717957	385.517975	0.421918	00:24
26	2761.279541	3091.217041	3074.397949	16.858736	-0.025576	0.628417	-0.979442	7453.478027	0.908650	441.615448	392.550476	0.421370	00:24
27	2730.678223	2962.628418	2947.173584	15.276114	-0.026963	0.620887	-1.002515	7304.295410	0.881647	428.941559	381.008789	0.418781	00:24
28	2711.719971	2964.291016	2948.281006	15.699995	-0.030957	0.629126	-0.976302	7354.960449	0.894260	436.180389	383.461853	0.418827	00:24
29	2698.613770	3050.802002	3035.411377	14.809456	-0.014705	0.621483	-1.001865	7517.462891	0.878165	427.194489	378.995758	0.423432	00:23
30	2680.392822	2954.147461	2937.375244	16.296328	-0.016200	0.636249	-0.954343	7354.759277	0.910993	443.337250	388.900665	0.419872	00:23
31	2648.716309	3001.396973	2986.461182	14.623203	-0.042457	0.618944	-1.007994	7513.009766	0.869564	422.928467	375.671539	0.415266	00:23
32	2552.094238	2815.679688	2799.725830	15.741020	-0.021439	0.625477	-0.987615	7107.967773	0.892987	435.489410	384.390106	0.415763	00:23
33	2500.885742	2701.386475	2685.672852	15.191346	-0.037839	0.645597	-0.924251	6860.041016	0.902515	438.233551	378.975342	0.412627	00:24
34	2422.822266	2719.418457	2703.131348	16.222771	-0.024178	0.639103	-0.945698	6888.277344	0.908967	442.192963	386.675140	0.419654	00:23
35	2360.017090	2604.458252	2588.693359	15.577497	-0.019458	0.639788	-0.944168	6656.376465	0.902958	438.855408	382.952759	0.422265	00:23
36	2323.858398	2629.382080	2613.721191	15.710530	-0.040843	0.643459	-0.932621	6623.768066	0.904269	438.465942	382.072937	0.420569	00:23
37	2305.494873	2616.138916	2600.468262	15.508165	-0.032108	0.642783	-0.933116	6720.474121	0.901037	437.298370	379.680634	0.414019	00:23
38	2282.103271	2587.460205	2572.022217	15.235188	-0.030462	0.647844	-0.918168	6601.239258	0.902206	437.860565	377.771149	0.416694	00:24
39	2269.184570	2588.785400	2572.932373	15.794830	-0.031420	0.647192	-0.920478	6507.833008	0.909731	442.258911	382.598511	0.418556	00:23

Using fastai V2: data pipelining

prep_df_for_datablocks[source]

get_ae_btfms[source]

get_ae_no_aug[source]

FastAI "block" for autoencoder

class TensorPoint[source]

class Tensor2Vect[source]

class LatentsTensor[source]

df_get_x[source]

df_get_y[source]

LatentsTensorBlock[source]

df_ae_x[source]

df_ae_y[source]

LatentTupleBlock[source]

get_ae_DataBlock[source]

creating the VAE

AE: deterministic AutoEncoder ( non- variational)

class UpsampleBlock[source]

class LatentLayer[source]

class AEEncoder[source]

class AEDecoder[source]

build_AE_encoder[source]

build_AE_decoder[source]

class AE[source]

Loss

class AELoss[source]

Metrics

class MyMetric[source]

class L1LatentReg[source]

helper for computing the KL Divergence as a function and a Module

class KLDiv[source]

class L2MeanMetric[source]

class L1MeanMetric[source]

class L2Metric[source]

class L1Metric[source]

class L2BMeanMetric[source]

class L1BMeanMetric[source]

class KLWeightMetric[source]

class RawKLDMetric[source]

class WeightedKLDMetric[source]

class MuMetric[source]

class MuSDMetric[source]

class StdMetric[source]

class StdSDMetric[source]

class LogvarMetric[source]

class LogvarSDMetric[source]

class AnnealedLossCallback[source]

default_KL_anneal_in[source]

parameter Splitters

bn_splitter[source]

resnetVAE_split[source]

AE_split[source]

default_AE_metrics[source]

other encoder and decoder types

get_conv_parts[source]

get_pretrained_parts[source]

get_encoder_parts[source]

constructing VAE with Module Class Layers

class VAELinear[source]

class VAELayer[source]

simple VAE class from VAE layer class components

get_pretrained_parts[source]

VAE Loss functions classes

class BVAELoss[source]

default_VAE_metrics[source]

"Vanilla" convolutional beta-VAE

MMD-VAE

class MMDVAE[source]

gaussian_kernel[source]

class MaxMeanDiscrepancy[source]

class MMDLoss[source]

class MMDMetric[source]

short_MMEVAE_metrics[source]

default_MMEVAE_metrics[source]

Vanilla MMD VAE

Higher alpha = 20

ResNet Encoder MMDVAE

L2 error func

resnet arch no freeze

vanilla MMD VAE

`prep_df_for_datablocks`[source]

`get_ae_btfms`[source]

`get_ae_no_aug`[source]

`class` `TensorPoint`[source]

`class` `Tensor2Vect`[source]

`class` `LatentsTensor`[source]

`df_get_x`[source]

`df_get_y`[source]

`LatentsTensorBlock`[source]

`df_ae_x`[source]

`df_ae_y`[source]

`LatentTupleBlock`[source]

`get_ae_DataBlock`[source]

`class` `UpsampleBlock`[source]

`class` `LatentLayer`[source]

`class` `AEEncoder`[source]

`class` `AEDecoder`[source]

`build_AE_encoder`[source]

`build_AE_decoder`[source]

`class` `AE`[source]

`class` `AELoss`[source]

`class` `MyMetric`[source]

`class` `L1LatentReg`[source]

helper for computing the KL Divergence as a function and a `Module`

`class` `KLDiv`[source]

`class` `L2MeanMetric`[source]

`class` `L1MeanMetric`[source]

`class` `L2Metric`[source]

`class` `L1Metric`[source]

`class` `L2BMeanMetric`[source]

`class` `L1BMeanMetric`[source]

`class` `KLWeightMetric`[source]

`class` `RawKLDMetric`[source]

`class` `WeightedKLDMetric`[source]

`class` `MuMetric`[source]

`class` `MuSDMetric`[source]

`class` `StdMetric`[source]

`class` `StdSDMetric`[source]

`class` `LogvarMetric`[source]

`class` `LogvarSDMetric`[source]

`class` `AnnealedLossCallback`[source]

`default_KL_anneal_in`[source]

parameter `Splitters`

`bn_splitter`[source]

`resnetVAE_split`[source]

`AE_split`[source]

`default_AE_metrics`[source]

`get_conv_parts`[source]

`get_pretrained_parts`[source]

`get_encoder_parts`[source]

`class` `VAELinear`[source]

`class` `VAELayer`[source]

`get_pretrained_parts`[source]

`class` `BVAELoss`[source]

`default_VAE_metrics`[source]

`class` `MMDVAE`[source]

`gaussian_kernel`[source]

`class` `MaxMeanDiscrepancy`[source]

`class` `MMDLoss`[source]

`class` `MMDMetric`[source]

`short_MMEVAE_metrics`[source]

`default_MMEVAE_metrics`[source]

`class` `UpsampleResBlock`[source]

`get_resblockencoder_parts`[source]

`class` `ResBlockAEDecoder`[source]

`build_ResBlockAE_decoder`[source]

`class` `ResBlockAE`[source]